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This study examined the impact of the Early Warning Intervention and Monitoring System (EWIMS), 
a systematic approach to the early identification of and intervention with students at risk of not 
graduating from high school on time. The study randomly assigned 73 schools to use EWIMS 
or to continue with their usual practices for supporting at-risk students. After a year of limited 
implementation, the study findings show that: 

EWIMS reduced chronic absence and course failure but not the percentage of students with 
low grade point averages or suspensions. 

EWIMS did not have a detectable impact on student progress in school (credits earned) or 
on school data culture — the ways in which schools use data to make decisions and identify 
students in need of additional support. 

The findings provide initial rigorous evidence that EWIMS is a promising strategy for reducing 
rates of chronic absence and course failure, two key indicators that students are off track for 
graduation. It is not clear what staff actions caused these improvements. EWIMS was challenging 
to implement in the first year and did not have an impact on other measured outcomes. 
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Summary 


Although high school graduation rates are rising — the national rate was 82 percent during 
the 2013/14 school year (U.S. Department of Education, 2015) — dropping out remains a 
persistent problem in the Midwest and nationally. Many schools now use early warning 
systems to identify students who are at risk of not graduating, with the goal of intervening 
early to help students get back on track for on-time graduation. Although research has 
guided decisions about the types of data and indicators used to flag students as being at risk, 
little is known about the impact of early warning systems on students and schools — and in 
particular, whether these systems do help get students back on track. This study, designed 
in collaboration with the REL Midwest Dropout Prevention Research Alliance, examined 
the impact and implementation of one early warning system — the Early Warning Inter- 
vention and Monitoring System (EWIMS) — on student and school outcomes. 

EWIMS is a systematic approach to using data to identify students who are at risk of not 
graduating on time, assign students flagged as at risk to interventions, and monitor at-risk 
students’ response to intervention. The EWIMS model provides schools with guidance to 
implement a seven-step process, supported by the use of an early warning data tool. The 
tool uses validated indicators, based on prior research, to flag students who are at risk of 
not graduating on time (Heppen & Therriault, 2008; Therriault, Heppen, O’Cummings, 
Fryer, & Johnson, 2010) and allows schools to assign students to interventions and monitor 
their progress. The indicators used to flag at-risk students in the tool are chronic absence 
(missed 10 percent of instructional time or more), course performance (failed any course, 
grade point average [GPA] below 2.0), behavioral problems (suspended once or more), and 
an off-track indicator (failed two or more semester-long or three or more trimester-long 
core courses or accumulated fewer credits than required for promotion to the next grade). 1 
The EWIMS model is intended to help schools efficiently use data to identify at-risk stu- 
dents and provide targeted supports. 

To assess the impact of EWIMS on student and school outcomes, 73 high schools in 
three Midwest Region states were randomly assigned to implement EWIMS during the 
2014/15 school year (37 EWIMS schools) or to continue their usual practices for identi- 
fying and supporting students at risk of not graduating on time and to delay implementa- 
tion of EWIMS until the following school year (36 control schools). The study included 
37,671 students in their first or second year of high school, with 18,634 students in EWIMS 
schools and 19,037 students in control schools. EWIMS and control schools and students 
were similar on all background characteristics prior to random assignment. 

The study examined the impacts of EWIMS on indicators of student risk and on student 
progress in school after the first year of EWIMS adoption. 

The study found that EWIMS reduced the percentage of students with risk indicators 
related to chronic absence and course failure but not related to low GPAs or suspension: 

• The percentage of students who were chronically absent (missed 10 percent or more 
of instructional time) was lower in EWIMS schools (10 percent) than in control 
schools (14 percent); this 4 percentage point difference was statistically significant. 

• The percentage of students who failed one or more courses was lower in EWIMS 
schools (21 percent) than in control schools (26 percent); this 5 percentage point 
difference was statistically significant. 


• The percentage of students who had a low GPA (2.0 or lower) was 17 percent in 
EWIMS schools and 19 percent in control schools; this difference was not statis- 
tically significant. However, sensitivity analyses that used continuous GPA data 
instead of the binary risk indicator showed that, on average, GPAs were higher in 
EWIMS schools (2.98) than in control schools (2.87); this difference was statisti- 
cally significant. 

• The percentage of students who were suspended once or more was 9 percent in 
both EWIMS and control schools; there was no statistically significant difference. 
EWIMS did not have an impact on student progress in school. That is, there was 
not a statistically significant difference between EWIMS and control schools in 
the percentage of students who earned insufficient credits to be on track to gradu- 
ate within four years (14 percent in both). 

At the school level, EWIMS did not have a detectable impact on school data culture, that 
is, the ways in which schools use data to make decisions and identify students in need of 
additional support. 

In nearly all participating schools, overall implementation of the EWIMS seven-step 
process was low, and implementation was challenging. Nevertheless, EWIMS schools were 
more likely than control schools to report using an early warning system and having a ded- 
icated team to identify and support at-risk students, but EWIMS schools did not differ from 
control schools in the frequency of data review or the number and type of interventions 
offered. 

This report provides rigorous initial evidence that even with limited implementation 
during the first year of adoption, using a comprehensive early warning system can reduce 
the percentage of students who are chronically absent or who fail one or more courses. 
These short-term results are promising because chronic absence and course failure in 
grades 9 and 10 are two key indicators that students are off track for on-time graduation. 
However, because the past research linking indicators to on-time graduation is correlation- 
al, it is not yet known if improving these indicators leads to improving on-time graduation 
rates. Also, EWIMS did not have a detectable impact on other measured indicators that 
are related to students’ likelihood of on-time graduation, including low GPAs, suspensions, 
and earning insufficient credits. 

Future research is needed to better understand the mechanisms through which EWIMS 
had an impact on chronic absence and course failure and why EWIMS did not affect other 
outcomes. In particular, studies could focus on identifying which staff actions and student 
experiences lead to improved student outcomes. Studies should also examine whether 
schools achieve improved overall implementation in subsequent years and whether (and 
how) the observed impacts fade, grow larger, or extend to other risk indicators (low GPAs 
and suspensions); to intermediate outcomes (including student persistence and progress in 
school); and to long-term outcomes (including dropout and on-time graduation rates). 
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Why this study? 


The national high school on-time graduation rate reached its highest level in U.S. history 
— 82 percent — during the 2013/14 school year (U.S. Department of Education, 2015). Even 
so, nearly one in five students did not graduate from high school, and graduation rates were 
lower for historically disadvantaged students. The most recent national graduation statistics 
also show that 73 percent of Black students and 76 percent of Hispanic students graduated 
from high school, compared with 87 percent of their White peers (U.S. Department of 
Education, 2015). Additionally, 75 percent of students from low-income families graduated 
in four years, as did 63 percent of English learner students and 63 percent of students in 
special education (U.S. Department of Education, 2015). The graduation rate was lower for 
male students (78 percent) than for female students (85 percent; Stetser & Stillwell, 2014)- 2 

The consequences of not graduating from high school are severe. When compared with 
graduating peers, students who drop out of school are more likely to be unemployed or 
underemployed, live in poverty, have poor health, and become involved in criminal activ- 
ities (Belfield & Levin, 2007; Christie, Jolivette, & Nelson, 2007; Hayes, Nelson, Tabin, 
Pearson, & Worthy, 2002), suggesting that increasing on-time graduation rates would 
benefit both individuals and society. 

States, districts, and schools are increasingly interested in using early warning systems to identify 
students who are at risk of not graduating on time and get them back on track 


One strategy 
for improving 
graduation rates 
is early warning 
systems, which 
use research- 
based warning 
signs to identify 
students at risk of 
not graduating 


Early warning systems have emerged as one strategy for improving graduation rates. Such 
systems use research-based warning signs to identify students at risk of not graduating. 3 
These warning signs can include indicators of engagement (for example, attendance), 
behavior (for example, suspensions), and course performance (for example, grades and 
credits) during middle and high school (Allensworth & Easton, 2005, 2007; Balfanz, 
Herzog, & Mac Iver, 2007; Neild & Balfanz, 2006; Silver, Saunders, & Zarate, 2008). 
More robust, comprehensive early warning systems also emphasize matching and assign- 
ing identified students to interventions to help them get on track for on-time graduation 
(Heppen & Therriault, 2008; Kennedy & Monrad, 2007; Jerald, 2006; Neild, Balfanz, & 
Herzog, 2007; Pinkus, 2008), as well as monitoring students’ progress in these interventions 
(O’Cummings, Heppen, Therriault, Johnson, & Fryer, 2010; O’Cummings, Therriault, 
Heppen, Yerhot, & Hauenstein, 2011). 

Educators have become increasingly interested in using early warning systems to identify 
students who are at risk of dropping out of school (Heppen & Therriault, 2008; Kennedy 
& Monrad, 2007; Neild et al., 2007). However, despite widespread implementation, there 
is little rigorous evidence of the impact of early warning systems on outcomes such as 
chronic absence, course failure, suspensions, progress in school, and, ultimately, on-time 
graduation. One recent experimental study tested the impact of Diplomas Now, a compre- 
hensive school reform strategy with more targeted interventions for students who display 
early warning signs, on indicators related to attendance, behavior, and course performance 
(Corrin, Sepanik, Rose, & Shane, 2016). The study, which focused on students in grades 6 
and 9, found that Diplomas Now had a positive and statistically significant impact on the 
percentage of students not flagged on any indicator but did not have a significant impact on 
average attendance, discipline, or course passing rates in either grade. Even with this new 
evidence of the limited impact of one type of early warning system on student indicators of 


1 


risk, there is not much information on the impact of adopting other early warning indica- 
tor models on student outcomes or school outcomes, including data culture. 4 

Members of the Midwest Dropout Prevention Research Alliance sought evidence of the 
impact of early warning systems on students and schools as a means to justify the costs 
associated with implementing them. To produce this evidence, the Regional Educational 
Laboratory (REL) Midwest and the Alliance collaborated on an experimental study of 
the impact of the Early Warning Intervention and Monitoring System (EWIMS) in 73 
high schools across three states. The intended audience for this report includes alliance 
members, practitioners, policymakers, researchers, and education decisionmakers consider- 
ing investing in an early warning system like EWIMS. 


The Early Warning Intervention and Monitoring System is a systematic approach to reliably 
identifying students at risk of not graduating on time, assigning them to interventions, and 
monitoring their progress 

EWIMS was developed by the U.S. Department of Education-funded National High School 
Center at American Institutes for Research. EWIMS is a systematic approach to identifying 
students at risk of not graduating on time, assigning them to interventions, and monitoring 
their progress, with the goal of getting at-risk students back on track for on-time gradua- 
tion. Schools implementing EWIMS receive guidance and site-based support to implement 
a seven-step process, which includes use of an early warning data tool (figure 1 and box 
1). Typical implementation of EWIMS includes on-site and virtual support from technical 
assistance staff, some of whom are former educators or researchers in dropout prevention 
strategies. Appendix A includes more information about the technical assistance liaisons 
and the implementation support they provided to EWIMS schools in this study. 


EWIMS is a 
systematic 
approach to 
identifying students 
at risk of not 
graduating on time, 
assigning them to 
interventions, and 
monitoring their 
progress, with the 
goal of getting 
at-risk students 
back on track for 
on-time graduation 


Figure 1. The Early Warning Intervention and Monitoring System seven-step 
implementation process 



Source: Early Warning Intervention and Monitoring System (EWIMS) Implementation Guide. For more infor- 
mation about EWIMS implementation, see http://www.earlywarningsystems.org/wp-content/uploads/docu- 
ments/EWSHSImplementationguide2013.pdf or Therriault et al. (2010). 
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Box 1. Early Warning Intervention and Monitoring System seven-step process and team 

The seven-step EWIMS process guides educators to use data to identify students who show warning signs of falling 
off track toward on-time graduation and to monitor students’ progress (see figure 1). Typical implementation of the 
model prioritizes identifying off-track students early in high school. The EWIMS steps are intended to be cyclical. 

Step 1 — Establish roles and responsibilities. Schools establish a team to lead and carry out the EWIMS process, 
determine the frequency and duration of meetings, and develop a shared vision for the team’s work. The EWIMS 
team may be newly established or may build on or be integrated into an existing team (for example, a school improve- 
ment team, response to intervention team, or student support team). According to the EWIMS model, the team 
should include a broad representation of staff within the school and, ideally, the district (for example, principals, 
teachers, district administrators, and counselors), and EWIMS activities should be a priority of the team. Because 
EWIMS implementation is aligned with the academic calendar, the EWIMS team is expected to meet monthly and 
examine students’ risk status and progress in interventions at the end of each grading period and at the end of the 
school year. 

Step 2 — Use the early warning data tool. The EWIMS team, with support from data or technology specialists, 
imports student demographic data and initial data on absences, course failure, grade point average, and behavior 
indicators into the early warning data tool (see box 2); updates administrative data as appropriate over the course 
of the school year; imports a list of available interventions into the tool; and runs automated or customized lists and 
reports. 

Step 3 — Review the early warning data. The EWIMS team focuses its attention on student- and school-level data, 
based on the indicators available in the tool. Data are reviewed to identify students who are at risk for not gradu- 
ating on time and to examine patterns in student engagement and academic performance within the school. This 
step is critical when using any type of early warning data, although the focus here is on using the “research-based” 
indicators and thresholds preloaded into the tool. Step 3 is revisited any time new data become available. 

Step 4 — Interpret the early warning data. The EWIMS team seeks out and brings in additional data (besides the 
indicators) to better understand the specific needs of individual students or groups of flagged students. Unlike step 
3, which is focused on the risk indicators in the tool, this step focuses on the underlying causes that might lead stu- 
dents to be identified as at risk on one or more indicators, using additional formal data (for example, administrative 
records) and informal input (for example, from teachers, family, and students). 

Step 5 — Assign and provide interventions. EWIMS team members make decisions about matching individual stu- 
dents to specific interventions in the school, district, and community, which are locally determined. 

Step 6 — Monitor students and interventions. The EWIMS team examines the student risk indicators on an ongoing 
basis to monitor the progress of students who have already been assigned to interventions. If these students contin- 
ue to be flagged as at risk, the EWIMS team may consider assigning them to different interventions; if some of these 
students are no longer at risk, the team may consider ramping down services. In the long term, schools also may 
alter their catalog of interventions based on their effectiveness (adding new interventions and dropping those that 
do not help students get back on track). This step provides critical ongoing feedback about additional student- and 
school-level needs and apparent successes. 

Step 7 — Evaluate and refine the early warning process. Through active and structured reflection, EWIMS team 
members revise specific strategies or their general approach as needed and determine how best to allocate resourc- 
es to support at-risk students. This step encourages EWIMS teams to make course corrections to any aspect of 
EWIMS implementation. As illustrated by the cyclical depiction of the seven-step process, this step (as well as the 
other six) reflects an ongoing process of continuous improvement. 
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Box 2. The early warning data tool 


The EWIMS model includes an early warning data tool that enables schools to routinely 
examine indicators of whether students are “off track" and take action, if warranted. Schools 
first import student-level data, a course catalog, and a list of all interventions available to 
students. The tool then automatically flags students as at risk using thresholds based on prior 
research (see Heppen & Therriault, 2008; Therriault et al., 2010). The indicators include the 
following: 1 

• Chronic absence flag. Missing 10 percent or more of instructional time (one flag for the first 
20 or 30 days, one flag per grading period, and a cumulative flag for the year). 

• Course failure flag. Failed one or more semester-long or trimester-long courses in any 
subject (one flag per grading period and a cumulative flag for the year). 

• Low grade point average flag. Earned a 2.0 or lower on a 4.0 scale or the equivalent on a 
different scale (one flag per grading period and a cumulative flag for the year). 

• Behavior flag. Suspended once or more, or flagged according to some other locally validat- 
ed definition (one flag per grading period and a cumulative flag for the year). 

• “Off track” flag. Failed two or more semester-long or three or more trimester-long core 
courses (math, science, English, and social studies) or accumulated fewer credits than 
required for promotion to the next grade (one cumulative flag for the year). The “off track” 
flag definition is based on Allensworth and Easton’s (2005; 2007) work on the “on-track” 
indicator. 

The tool allows schools to customize settings (for example, by creating their own flag for 
students who failed grade 9 algebra), group students in various ways, and produce reports 
(including individual and student- and school-level data summaries) to guide dropout preven- 
tion strategies. The tool also allows and encourages users to record the assignment of flagged 
students to available interventions and monitor students’ response to those interventions. 

Note 

1. The early warning data tool also includes an "incoming risk” flag, but schools in the study did not use it 
systematically. See appendix A for more detail on the incoming risk flag and how it was used in this study. 


The early warning 
data tool flags 
students at risk 
using indicators 
drawn from 
prior research 
on the strongest 
predictors 
of on-time 
graduation. The 
tool allows schools 
to assign students 
to interventions 
and monitor 
their progress 
through multiple 
reporting features 


The early warning data tool flags students at risk using indicators drawn from prior research 
on the strongest predictors of on-time graduation (see Heppen & Therriault, 2008; Them- 
ault et al., 2010; box 2). In addition to flagging at-risk students, the tool allows schools 
to assign students to interventions and monitor their progress through multiple reporting 
features. The EWIMS model is intended to systematically and continually improve the 
ways that schools use data to identify at-risk students and efficiently and effectively provide 
targeted supports. EWIMS does not prescribe specific interventions; instead, it encourages 
schools to inventory their available interventions and consider (as part of the seven-step 
process) which are best suited to address at-risk students’ needs. 

The Early Warning Intervention and Monitoring System is expected to improve student- and school- 
level outcomes 

The theoretical framework describes how EWIMS is expected to improve student and 
school outcomes (figure 2). EWIMS is intended to focus and streamline the data review 
process by using research-based early warning indicators to flag students who may be at 
risk of not graduating on time. This, it is assumed, will allow schools to more systemati- 
cally identify students who need support. A dedicated team to identify and support at-risk 
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Figure 2. Theory of action for how the Early Warning Intervention and Monitoring 
System improves student and school outcomes 
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EWIMS is the Early Warning Intervention and Monitoring System. 
Source: Authors' elaboration of the theory of action. 


The effectiveness 
of EWIMS for 
students generally 
depends on 
the quality and 
appropriateness 
of the support 
provided 


students (the EWIMS team) can then use this information to better align the type of 
support to specific students’ needs. The effectiveness of EWIMS for students, therefore, 
depends on the quality and appropriateness of the support provided. 


The use of EWIMS is expected to have short-term impacts on both schools and students. 
At the school level, EWIMS implementation is expected to change how schools use data 
to identify and support at-risk students, leading to improvements in some aspects of school 
data culture: for example, improvements in the context for data use (for example, goals and 
professional climate for data use), concrete supports for data use (for example, allocated 
time for using data or professional development on data use), data-driven student support 
(for example, data-based decisions about how to best target limited supports for students), 
and reduced barriers to data use (for example, lack of time to review data). Other aspects 
of school data culture (for example, professional climate for data use) may require several 
years to show improvement. 


At the student level, EWIMS implementation should result in short-term reductions in 
the prevalence of students being flagged by indicators related to chronic absence (missing 
10 percent or more instructional time), course failure (one or more course failures, GPAs 
of 2.0 or lower), and behavioral problems (for example, suspensions). These short-term 
reductions are then expected to lead to improved intermediate outcomes, including 
improvements in students’ progress in school (by earning sufficient credits to remain on 
track toward on-time graduation) and persistence in school (by remaining continuously 
enrolled). Over the long term, EWIMS schools should see improved on-time graduation 
rates as a result of improvements in students’ progress and persistence. 


What the study examined 


Together, the REL Midwest and the Dropout Prevention Alliance collaborated to design 
and conduct a randomized controlled trial to examine the early impact of EWIMS on 
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student and school outcomes. The study examined the following research questions about 
the impact of EWIMS a year after its adoption: 

1. What is the impact of EWIMS on indicators of student risk? 

2. What is the impact of EWIMS on student progress in school? 


The indicators of student risk were binary, meaning that they indicate whether students 
were above or below the thresholds used as the default settings in the early warning data 
tool; specifically, whether they missed 10 percent or more of instructional time, failed one 
or more courses, had GPAs of 2.0 or lower, and had one or more suspensions. Student 
progress was also binary: whether or not students had earned sufficient credits to be on 
track to graduate within four years (defined as earning one-fourth of the credits needed 
to graduate for first-year students and one-half of the credits needed to graduate for sec- 
ond-year students). 

The study also examined whether the impact of EWIMS differed for first- and second-year 
students, because typical implementation of the model prioritizes identifying at-risk stu- 
dents as early in high school as possible (that is, the focus of early implementation is often 
on students in grade 9). In addition, the study posed an exploratory research question 
about the impact of EWIMS on school data culture, a key school-level outcome in the 
EWIMS theory of action. This question was considered exploratory because the study was 
not designed to detect significant impacts on school-level outcomes. 


The study was 
a snapshot of 
early adoption 
of EWIMS and 
was not designed 
to examine 
implementation 
and student 
progress over 
multiple years 


The study was a snapshot of early adoption of EWIMS and was not designed to examine 
implementation and student progress over multiple years. Therefore, persistence and 
dropout across school years and on-time graduation could not be examined but are critical 
outcomes for future research. 


Four research questions about implementation were examined to provide context for 
understanding the impact of EWIMS on the main study outcomes: 

1. To what extent did EWIMS schools participate in the professional development 
provided and implement the EWIMS seven-step process? 

2. What barriers to implementation did EWIMS schools experience? 

3. What types of interventions did EWIMS schools provide to students identified as 
at risk, and what percentage of students received those services? 

4- To what extent did EWIMS and control schools differ in their practices for identi- 
fying and supporting students at risk of not graduating on time? 

The study addressed these questions about EWIMS impact and implementation using 
a randomized controlled trial and quantitative and qualitative data. (Box 3 provides a 
summary of the data and methods used, and appendixes B and C provide more details.) 

A total of 73 schools in three Midwest Region states participated in the study. 5 The 
schools were randomly assigned to either the treatment condition, with schools imple- 
menting EWIMS from spring 2014 through the end of the 2014/15 school year (37 EWIMS 
schools), or to the control condition (36 control schools). The control schools continued 
their usual practices for identifying and supporting students at risk of not graduating on 
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time during the 2014/15 school year and were provided EWIMS in the following school 
year (2015/16). The study included 37,671 students in their first or second year of high 
school, with 18,634 students in EWIMS schools and 19,037 students in control schools 
(see table B3 in appendix B). First-year students were enrolled in grade 9 in the 2014/15 
school year, and second-year students were enrolled in grade 9 in the previous (2013/14) 
school year. Differences between EWIMS schools and control schools were not statistically 
significant on any measured baseline characteristics (see table B5). 


Box 3. Study design, data, and methods 
Study design 

This study used a randomized controlled trial to examine the impact of EWIMS on student 
and school outcomes. Schools were matched into pairs within states and districts based on 
school size, graduation rates, and initial dropout prevention efforts. Next, schools were ran- 
domly assigned within each pair to either implement EWIMS during the 2014/15 school year 
(37 EWIMS schools) or to continue their usual practices for identifying and supporting students 
at risk of not graduating on time and implement EWIMS in the following school year (36 control 
schools). See appendix B for details on the design, sample, and random assignment. 

Data collection 

The following data were collected for all schools (see appendix C for further details): 

• Extant student records from the 2012/13 school year through spring 2015. 

• School leader responses to a web-based survey administered in spring 2015 to measure 
school data culture and collect information about interventions used to support at-risk stu- 
dents. The survey was also administered in spring 2014 (after random assignment), but was 
used only as an additional data source to identify interventions available in EWIMS schools. 

The following data were collected only for EWIMS schools (see appendix C for further details): 

• Extant documents on EWIMS implementation during the 2014/15 school year. 

• Monthly logs of the content and frequency of EWIMS team meetings during the 2014/15 
school year. 

• Reports from the early warning data tool that measured tool use through spring 2015. 

• Interviews with EWIMS team members conducted in spring 2015. 1 

Measures 

Student outcome measures. The student outcomes measures for the four risk indicators (missed 
10 percent or more of instructional time, failed one or more courses, GPA of 2.0 or lower, and one 
or more suspensions) and for student progress in school were binary variables. Each binary vari- 
able was coded 1 or 0, reflecting whether the student was above or below the threshold for each 
risk indicator, or for progress in school, whether the student had earned sufficient credits to be on 
track to graduate within four years. See appendix C for operational definitions of each outcome. 

School data culture measures. School data culture was measured with a set of survey items 
on the 2015 end-of-year school leader survey. These items yielded an overall score for data 
culture and subscores for four key dimensions: context for data use, concrete supports for 
data use, data-driven student support, and barriers to data use (table C3). 

EWIMS implementation measures. Measures of school participation in each of the EWIMS 
professional development sessions — regional trainings, tool trainings, online trainings (called 

(continued) 
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Box 3. Study design, data, and methods (continued) 


WebShares), and school site visits — were based on attendance records indicating which 
school staff attended the sessions. Levels of implementation of the seven steps of the EWIMS 
process were generated using a rubric developed for the study. Measures of barriers to EWIMS 
implementation and specific types of interventions offered in EWIMS schools were extracted 
from extant records, surveys, and interviews and coded with key themes. Additional measures 
were used to assess the contrast between EWIMS schools and control schools in their prac- 
tices for identifying and supporting at-risk students. These measures included the frequency 
of data review, the number and type of interventions, whether schools reported using an early 
warning system, and whether schools reported having a dedicated school-based team or group 
of individuals that reviews student data to support students identified as at risk of not graduat- 
ing from high school. See appendix C for further detail. 


Impact analysis 

Multilevel logistic and linear regression models with students nested in schools were used to 
estimate the impact of EWIMS on student outcomes for the main research questions. Student- 
level covariates (level 1) and fixed effects for matched pairs (level 2) were included in these 
models to increase the precision of the estimate of the impact of EWIMS at both levels. Sen- 
sitivity analyses were conducted to determine if the impact of EWIMS was robust to different 
model specifications and whether the results were similar when the binary outcomes were 
replaced with their continuous counterparts. For example, low GPA (2.0 or lower) was replaced 
with GPA. See the “Impact analyses” section in appendix C for more information on the analyt- 
ic approach and tables D1 and D2 in appendix D for sensitivity analysis findings. 


Implementation analysis 

To address implementation research questions, descriptive analyses of implementation data 
were conducted. Treatment contrast analyses used linear and logistic regression models with 
school covariates that tested whether or not EWIMS and control schools differed in their prac- 
tices for identifying and supporting at-risk students. See appendix C, pages C-16-C-22, for 
more detail on the implementation analyses. 

Note 

1. Exit interviews were conducted with schools that chose to stop implementing EWIMS during the 2014/15 
school year. See appendix C for further details on the interview and analytic approach; see appendix D for 
detailed findings. 


Differences 
between EWIMS 
schools and control 
schools were 
not statistically 
significant on any 
measured baseline 
characteristics 


What the study found 


This section presents the main study findings for the impact of EWIMS on student and 
school outcomes and documents the implementation of EWIMS in study schools. 

The Early Warning Intervention and Monitoring System reduced the percentage of students with risk 
indicators related to chronic absence or course failure but did not have a detectable effect on the 
percentage who had a low grade point average or were suspended 

After one year of implementation, EWIMS reduced the percentage of students who were 
chronically absent or failed one or more courses but did not have an impact on the per- 
centages of students who had a low GPA or were suspended (figure 3). Sensitivity analyses 
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show that the findings reported here are robust and consistent across different analytic 
approaches (see tables D1 and D2 in appendix D). 


Chronic absence. The percentage of students who were chronically absent (that is, missed 
10 percent or more of instructional time) was lower in EWIMS schools (10 percent) than 
in control schools (14 percent; see figure 3). This 4 percentage point difference was statis- 
tically significant. The impact of EWIMS on chronic absence was larger for first-year stu- 
dents than for second-year students (see figure D1 and table D3 in appendix D). Sensitivity 
analyses that used continuous data on instructional time missed (instead of the binary risk 
indicator) showed that the average percentage of instructional time missed was statistically 
significantly lower in EWIMS schools (5.4 percent) than in control schools (6.5 percent; 
see table D2). 

Course failure. The percentage of students who failed one or more courses was lower 
in EWIMS schools (21 percent) than in control schools (26 percent; see figure 3). This 
5 percentage point difference was statistically significant. The impact of EWIMS on 
course failure was larger for first-year students than for second-year students (see figure 
D1 and table D3 in appendix D). Sensitivity analyses that used continuous data instead 
of the binary risk indicator showed that the average percentage of courses that students 
failed (out of the number of courses attempted) was also statistically significantly lower 


EWIMS reduced 
chronic absence 
and course 
failure but not 
the percentage 
of students with 
low grade point 
averages or 
suspensions 


Figure 3. The Early Warning Intervention and Monitoring System reduced the 
percentage of students with risk indicators related to chronic absence and course 
failure but not the percentage with indicators related to low GPA or suspensions in 
2014/15 



*** difference significant at p < .001. 

EWIMS is the Early Warning Intervention and Monitoring System. GPA is grade point average. 

Note: Model-adjusted percentage of students identified as at risk in EWIMS and control schools, controlling for 
school and student covariates, are presented. Higher values indicate a larger percentage of students at risk. 
Sample included 65 schools and 35,876 students for "chronic absence”; 65 schools and 35,133 students for 
“failed any course”; 57 schools and 30,080 students for "low GPA’’; and 63 schools and 35,501 students for 
“suspended." Note that less than 1 percent of the student analytic sample was dropped for chronic absence, 
low GPA, and suspended due to perfect prediction. Additional details about the models and samples used to 
generate these findings can be found in the notes to table D1 in appendix D. 

Source: Authors' analysis based on extant student records from schools, school districts, and state education 
agencies described in appendix C. 
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in EWIMS schools (8 percent) than in control schools (10 percent; see table D2). Also, 
the percentage of students who failed one or more core academic courses (English, math, 
science, and social studies) during the 2014/15 school year was lower in EWIMS schools 
(20 percent) than in control schools (24 percent) — a 4 percentage point difference that 
was statistically significant (see table Dl). 

Low grade point average. The percentage of students who had a GPA of 2.0 or lower was 
17 percent in EWIMS schools and 19 percent in control schools (see figure 3). This dif- 
ference was not statistically significant (see table Dl in appendix D). However, sensitivity 
analyses that used continuous GPA data instead of the binary risk indicator showed that, 
on average, GPAs were higher in EWIMS schools (2.98) than in control schools (2.87); 
this difference was statistically significant (see table D2). 

Suspension. The percentage of students who were suspended once or more was 9 percent 
in both EWIMS and control schools, and the difference was not statistically significant 
(see figure 3, and table Dl in appendix D). 6 

The Early Warning Intervention and Monitoring System did not have a detectable impact on student 
progress in school 

There was no statistically significant difference in the percentage of students who, by the 
end of the 2014/15 school year, had earned insufficient credits to be on track to graduate 
within four years. The percentage of students with insufficient credits was 14 percent in 
both EWIMS and control schools (see table Dl in appendix D). Sensitivity analyses that 
used continuous credits earned instead of the binary risk indicator were consistent; that 
is, there was no statistically significant difference in the average number of credits earned 
between EWIMS and control schools (students earned an average of 13 credits in both; see 
table Dl). 

As noted earlier, it was out of scope for this study to examine persistence or dropout across 
school years. However, analysis of a preliminary measure of persistence within the 2014/15 
school year indicated that 95 percent of the students in both EWIMS and control schools 
were still enrolled at the end of the 2014/15 school year and the difference was not statisti- 
cally significant. See appendix C and figure D2 and table D4 in appendix D for more detail 
about the measure and analysis of preliminary persistence. 

The Early Warning Intervention and Monitoring System did not have a detectable impact on school 
data culture 

EWIMS did not have a detectable impact on school data culture, as measured with the 
2015 end-of-year survey of school leaders (figure 4). Differences between EWIMS schools 
and control schools on the overall data culture scale or any of its subscales, including 
context, concrete supports, barriers for data use, and data-driven student support, were not 
statistically significant (see table D5 in appendix D). However, the effect size (Hedges’ g) 
for the overall school data culture scale was 0.27, suggesting that although not statistically 
significant, EWIMS schools reported modestly higher data culture than control schools 
(see table D5). 7 As noted earlier, analyses of school-level outcomes are considered explor- 
atory because the study did not include a large enough number of schools to detect modest 
effects on school-level outcomes. 


The Early Warning 
Intervention 
and Monitoring 
System did not 
have a detectable 
impact on student 
progress in school 
or on school 
data culture 
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Figure 4. The Early Warning Intervention and Monitoring System did not have a 
detectable impact on school data culture at the end of the 2014/15 school year 



Overall Context 

data culture 


Concrete Data-driven Lack of 

supports student support barriers 


EWIMS is the Early Warning Intervention and Monitoring System. 

Note: Sample included 66 schools that completed the school leader survey items (32 EWIMS schools and 34 
control schools) for overall data culture, concrete supports, data-driven student support, and lack of barri- 
ers. Sample included 67 schools that completed the school leader survey items (33 EWIMS schools and 34 
control) for context. Data culture items were measured on a scale of 1 to 4, with 1 being low data culture and 
4 being high data culture. The items that compose the scale for barriers to data use were reverse coded, such 
that a higher score indicated fewer barriers. The differences between the EWIMS and control schools in stan- 
dard deviation units (Hedges’ g, using a pooled standard deviation) were 0.27 for overall data culture, -0.02 
for context, 0.22 for concrete supports, 0.31 for data-driven student support, and 0.19 for lack of barriers. 
Regression models that regressed data culture on treatment status, a set of three covariates (school size, 
baseline graduation rate, and baseline data-driven dropout prevention efforts), and a set of variables capturing 
school matched pairs revealed no statistically significant differences at the p < .05 level. Additional details 
about these findings can be found in table D5 in appendix D. 

Source: Authors' analysis based on school leader survey administered in spring 2015. 


For participating schools, the level of overall implementation of the Early Warning Intervention and 
Monitoring System seven-step process was low, and implementation was challenging 

Despite the training and support that EWIMS schools received, the implementation 
findings suggest that schools found it difficult to implement the model in the first year 
of adoption. Approximately 80 percent of EWIMS schools implemented EWIMS as 
planned in the 2014/15 school year. Out of the full sample of 37 EWIMS schools, one 
never implemented EWIMS (and dropped out of the study after random assignment, but 
before EWIMS implementation began) and seven stopped implementing EWIMS during 
the 2014/15 school year. 8 The sections that follow summarize information on school par- 
ticipation in EWIMS training, levels of implementation for each of the seven steps and 
overall, barriers to implementation experienced by EWIMS schools, and the specific types 
of interventions offered in EWIMS schools and the percentage of students who received 
those services. 


Despite the 
training and 
support that 
EWIMS schools 
received, the 
implementation 
findings suggest 
that schools 
found it difficult 
to implement the 
model in the first 
year of adoption 


Participation in training on the early warning data tool and seven-step process was high 
among EWIMS schools at the start but declined during the 2014/15 school year. EWIMS 
implementation liaisons delivered a total of 11 trainings to EWIMS schools between April 
2014 and June 2015. These included an individual school training on how to use the early 
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warning data tool, regional training (for multiple EWIMS schools) on how to implement the 
seven-step process, a refresher tool training at the beginning of the 2014/15 school year, and 
ongoing follow-up throughout the 2014/15 school year (including school site visits, online 
trainings called WebShares, and responsive technical assistance using telephone and email 
on an as-needed basis). Participation in EWIMS trainings declined throughout the 2014/15 
school year, from a high of 97 percent for the first regional training to a low of 59 percent for 
the fourth WebShare meeting (figure 5). However, throughout the 2014/15 school year, staff 
satisfaction with EWIMS trainings was high — more than 90 percent of respondents were 
either satisfied or very satisfied with each training (see table D6 in appendix D). 


Only two schools achieved moderate or high levels of implementation across all seven 
steps in the 2014/15 school year. EWIMS schools were categorized as high, moderate, 
or low implementers of the full EWIMS model based on a combination of multiple key 
features per step (see table D7 in appendix D). This measure was developed to gauge full 
implementation of EWIMS across all seven steps at any point of implementation, not 
just in the first year of adoption. Because EWIMS is intended to be a process of con- 
tinuous improvement across multiple years, achieving high ratings in the first year may 
be challenging for many schools. Higher levels of implementation might be expected in 
subsequent years as schools reflect on successes and challenges from their first year and 
make improvements. However, it is also possible that implementation levels might decline 
in subsequent years if schools lose interest or motivation to implement EWIMS; a longer 
study is needed to document implementation levels over time. 


Figure 5. Participation in professional development sessions was highest for the 
initial trainings and decreased for site visits and WebShares during 2014/15 

100 



During the first 
full school year of 
implementation, 
all but two EWIMS 
schools were 
categorized as 
low implementers 
across all seven 
steps. However, 
many EWIMS 
schools had 
moderate or high 
implementation 
ratings for 
individual steps 
of the seven-step 
model 


Note: The full Early Warning Intervention and Monitoring System (EWIMS) school sample includes 37 schools, 
one of which never implemented EWIMS (and dropped out of the study after random assignment, but before 
EWIMS implementation began) and seven of which stopped implementing EWIMS during the 2014/15 school 
year. Professional development sessions are presented in the order in which they were provided to EWIMS 
schools. A timeline of these activities can be found in table A1 in appendix A. 

Source: Authors' calculations based on attendance sheets collected during each professional development 
session. 
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During the first full school year of implementation (2014/15), all but two EWIMS schools 
(95 percent) were categorized as low implemented. 9 Across all seven steps, one school 
achieved a moderate implementation rating and one achieved a high implementation 
rating. However, many EWIMS schools had moderate or high implementation ratings for 
individual steps of the severvstep model (11 percent to 51 percent; figure 6 and see table D7 
in appendix D). 


Twenty-six EWIMS schools (70 percent) achieved high ratings on at least one step of the 
EWIMS process, and eight schools (22 percent) achieved high ratings on at least four of 
the seven steps (see table D8 in appendix D). More than a third (38 percent) of EWIMS 
schools were coded as being high implementers of step 1 (establishing roles and responsibih 
ities) and more than half (51 percent) as being high implementers of step 2 (using the early 
warning data tool). Steps 3-6 — reviewing and interpreting EWIMS data, assigning and 
providing interventions to students, and monitoring students over time — appeared more 
challenging for most EWIMS schools to implement at high levels in the 2014/15 school 
year; only 11 percent to 22 percent achieved high levels of implementation of these steps. 
However, almost half (49 percent) of EWIMS schools had a high level of implementation 
of step 7 (evaluating and refining the EWIMS process), suggesting that even with limited 
overall implementation across the full process, schools reflected on how they used EWIMS 
and either made changes to meet their needs throughout the school year or planned future 
changes to EWIMS for the following school year. 

Barriers experienced by schools implementing the Early Warning Intervention and 
Monitoring System in the 2014/15 school year included difficulty using the early 
warning data tool and staffing issues. Data from the school leader survey, interviews, and 


Schools 
experienced 
barriers to 
implementation 
such as technical 
difficulties 
uploading student 
data into the early 
warning data 
tool and changes 
in staffing that 
affected the 
EWIMS team 


Figure 6. Many Early Warning Intervention and Monitoring System schools achieved 
moderate and high implementation of individual steps during 2014/15 


■ Low b Moderate b High 


Step 1 — Establish roles and responsibilities 
Step 2 — Use the early warning data tool 
Step 3 — Review the early warning data 
Step 4 — Interpret the early warning data 
Step 5 — Assign and provide interventions 
Step 6 — Monitor students and interventions 
Step 7 — Evaluate and refine the early warning process 

Overall 



Percentage of schools 


Note: The full Early Warning Intervention and Monitoring System (EWIMS) school sample included 37 schools, 
one of which never implemented EWIMS and dropped out of the study after random assignment but before 
intervention, and seven of which stopped implementing EWIMS during the 2014/15 school year. Additional 
details about the findings presented in this figure can be found in table D7 in appendix D. 

Source: Authors' calculations based on school leader survey, early warning data tool reports, monthly meeting 
logs, and EWIMS team interviews. 


13 


documentation from the EWIMS technical assistance liaisons suggest that schools experi- 
enced notable barriers to implementing EWIMS in the 2014/15 school year. In particular, 
schools encountered difficulty importing data into the early warning data tool (24 schools, 
65 percent), sometimes as a result of incompatibility with student information systems (6 
schools, 16 percent), limited technical and data capacity of staff assigned to support tool 
use (5 schools, 14 percent), or limited personnel time to dedicate to importing data into the 
tool (13 schools, 35 percent). In addition, four schools (11 percent) experienced turnover 
of key staff, such as the principal or the individual responsible for preparing and importing 
data into the tool. Two schools (5 percent) preferred to use their own student information 
system to flag students at risk of not graduating on time (instead of the early warning data 
tool) but continued to implement the seven-step EWIMS process. 10 Implementation chal- 
lenges appeared to be insurmountable for the eight schools that stopped or never began 
implementing EWIMS during the study (22 percent of the EWIMS school sample; see 
page D-8 in appendix D). 

EWIMS schools offered a range of interventions to support at-risk students, but accord- 
ing to the data in their early warning data tools, less than 30 percent of the flagged 
students were assigned to interventions. Across all EWIMS schools with data in their 
early warning data tools, data for 19,309 students had been uploaded. 11 Of these students, 
50 percent (9,559) were flagged on at least one risk indicator during the 2014/15 school 
year: 30 percent for chronic absence, 26 percent for failing one or more courses, 24 percent 
for a low GPA, and 6 percent for suspensions. About 12 percent of students were flagged for 
both chronic absence and course failures. More detail about how these samples differ from 
those in the primary impact models is shared in appendix D, page D-9. 


The most 
common types 
of interventions 
offered in EWIMS 
schools were 
academic supports; 
attendance 
and behavioral 
supports were 
less common 


A key step in implementing EWIMS is that schools assign students to interventions and 
monitor their progress over time. On average, data from the early warning data tools indi- 
cate that EWIMS schools assigned 27 percent of flagged students to at least one interven- 
tion in the 2014/15 school year (ranging from 0 to 67 percent within schools). Moreover, 
22 percent of the 9,559 students identified as at risk in the early warning data tools were 
assigned to interventions aligned to their risk indicators (for example, students flagged for 
course failure were assigned to academic interventions). However, these analyses should be 
interpreted with caution because they rely on schools’ use of the intervention features in 
the early warning data tool, and assignment to an intervention may have occurred outside 
of the tool. 


The most common types of interventions offered in EWIMS schools were academic sup- 
ports; attendance and behavioral supports were less common (see table D9 in appendix 
D). Interventions ranged from formal programs (such as online credit recovery for students 
who failed a course) to less formal strategies (such as meeting with a student or parents). 
Twenty-six of the 37 EWIMS schools (70 percent) offered at least one academic inter- 
vention to support at-risk students. For example, 38 percent of EWIMS schools offered 
targeted supports in English language arts, and 35 percent offered targeted supports in 
algebra. In addition, 68 percent of EWIMS schools offered tutoring to students, of which 
19 percent offered peer tutoring. Nearly two- thirds of EWIMS schools (65 percent) offered 
credit recovery interventions, while a smaller subset of schools offered online credit recov- 
ery (27 percent). A majority of EWIMS schools (62 percent) offered mentoring programs. 
Fewer schools (30 percent) used peer mentors. Behavioral and attendance interventions 
were less common in EWIMS schools (24 percent of schools focused on attendance using 
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truancy interventions, 16 percent had interventions that focused primarily on behavior 
through disciplinary actions, and 14 percent had dedicated social emotional interven- 
tions). Additional nonacademic support intervention strategies included conferences with 
students and parents (41 percent of schools), letters or phone calls home (38 percent), 
counseling (30 percent), student contracts (24 percent), and mental and physical health 
services (24 percent; see table D9 in appendix D). 


Early Warning Intervention and Monitoring System schools were more likely than control schools to 
report using an early warning system and having a dedicated team to identify and support at-risk 
students but did not differ from control schools in the self-reported frequency of data review and 
number and type of interventions offered 


To examine the contrast between EWIMS and control schools in their practices related to 
identifying and supporting at-risk students, the study used data from the spring 2015 school 
leader survey. To gauge the extent to which schools adhered to their randomly assigned 
groups, the survey asked school leaders whether they used an early warning system during 
the 2014/15 school year (see appendix C for the definition of early warning systems pro- 
vided to school leaders during on-site or virtual presentations as part of the recruitment 
process for the study). Beyond self-reported use of an early warning system, the study also 
examined contrasts between EWIMS and control schools in some early warning system- 
related practices. These analyses included items asking schools whether they had a ded- 
icated school-based team or group of individuals that reviewed student data to support 
students identified as at risk of not graduating from high school (hereafter referred to as 
a dedicated team to identify and support at-risk students), how often they reviewed atten- 
dance and course performance data, and how many and what types of interventions they 
offered to students. 


To examine the 
contrast between 
EWIMS and control 
schools in their 
practices related 
to identifying and 
supporting at-risk 
students, the study 
used data from the 
spring 2015 school 
leader survey 


The results suggest that EWIMS and control schools generally reported adhering to 
random assignment — most EWIMS schools reported using an early warning system and 
most control schools did not. Of the five measures used to assess contrasts in specific prac- 
tices, EWIMS and control schools differed on one: having a dedicated team to identify 
and support at-risk students. On the other four measures, self-reported differences between 
EWIMS and control schools were not statistically significant. However, because the study 
was not designed to detect statistically significant differences on school-level measures, 
effect sizes for the magnitude of these differences are presented below. 

Has an early warning system. Consistent with the random assignment groupings, many 
more EWIMS schools than control schools reported using an early warning system (figure 
7). This difference was statistically significant and large in magnitude, translating to an 
effect size of 2.50 (see table DIO in appendix D). 

Has a dedicated team to identify and support at-risk students. More EWIMS schools 
than control schools reported having a dedicated team to identify and support at-risk stu- 
dents (see figure 7) — a key first step in the EWIMS seven-step process. This difference was 
statistically significant and large in magnitude, translating to an effect size of 0.95 (see 
table D10 in appendix D). 

Frequency of data review. School leaders from nearly all EWIMS and control schools 
(91 percent in both) reported that their schools reviewed both attendance and course 
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failure data to identify at-risk students during the 2014/15 school year (see table Dll in 
appendix D). Seventy-six percent of EWIMS schools reported that they reviewed atten- 
dance data at least monthly, compared with 88 percent of control schools (see figure 7). 
This difference was not statistically significant, although the effect associated with this 
difference was of notable size (-0.51), and favored control schools (see table D12). In con- 
trast, 53 percent of EWIMS schools reported reviewing course failure data at least monthly, 
compared with 42 percent of control schools. This difference was not statistically signifi- 
cant, and the effect size was 0.25 (see figure 7 and table D12). 


Number and type of interventions. There were no statistically significant differences 
between EWIMS and control schools in the number or type of interventions that school 
leaders reported they had available to support students. With respect to the number of 
interventions, EWIMS schools reported an average of 2.75 interventions and control 
schools reported an average of 2.20 interventions, a difference that translates to an 
effect size of 0.29 (see page D-12). With regard to types of interventions, few EWIMS or 


Figure 7. Early Warning Intervention and Monitoring System schools and control 
schools differed on some but not all self-reported early warning system-related 
practices during 2014/15 



Has an Has a dedicated team Reviews Reviews 

early warning to identify and support attendance data course failure data 

system at-risk students at least monthly at least monthly 


Of the five 
measures used to 
assess contrasts in 
specific practices, 
EWIMS and control 
schools differed 
on one: having a 
dedicated team to 
identify and support 
at-risk students. 

On the other 
four measures, 
self-reported 
differences 
between EWIMS 
and control 
schools were 
not statistically 
significant 


EWIMS is the Early Warning Intervention and Monitoring System. 

Note: The school sample includes the 66 schools (32 EWIMS schools and 34 control schools) that completed 
the school leader survey for the items about data review and early warning systems and the 65 schools (31 
EWIMS schools and 34 control schools) that completed the school leader survey for the item about the dedi- 
cated team to identify and support at-risk students. The five EWIMS schools that reported not using an early 
warning system were among the eight schools that had never started, or that stopped, implementing EWIMS 
in the 2014/15 school year. The four control schools that reported using an early warning system described 
those systems as involving behavior intervention monitoring and more informal systems, such as regular 
meetings among counselors and administrators. The item measuring whether or not a school has a dedicated 
team to identify and support at-risk students included an “other” response option; two EWIMS and four control 
schools responded with this response, indicating that the work was done by smaller teams or that schools 
were just putting the team together. Logistic regression models that regressed a binary indicator of whether 
or not a school had an early warning system, had a school-based team, reviewed attendance data monthly, 
or reviewed course failure data at least monthly on treatment status and a set of three covariates (school 
size, baseline graduation rate, and baseline data-driven dropout prevention efforts) revealed no statistically 
significant differences at the p < 0.05 level between EWIMS and control schools. Additional details about the 
findings presented in this figure can be found in tables DIO and D12 in appendix D. 

Source: Authors' calculations based on school leader survey. 
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control schools offered attendance or behavior interventions, but all control and nearly 
all EWIMS schools offered course performance (academic) interventions (see table D13 in 
appendix D). 


Implications of the study findings 


In 2008, the U.S. Department of Education’s Dropout Prevention Practice Guide listed 
“using data systems as a diagnostic tool to understand dropout trends and identify individ- 
ual students at risk of dropping out” as the first of six related recommendations (Dynarski 
et al., 2008). However, no rigorous evidence was available at the time to support this use of 
data systems. Nevertheless, schools, districts, and states across the country are increasingly 
using early warning systems to identify students at risk of not graduating on time. This 
study provided an initial large-scale, rigorous test of the use of this strategy, focusing specif- 
ically on the EWIMS model. 

Despite low levels of implementation, the study found that EWIMS reduced the per- 
centage of students who were flagged by risk indicators related to chronic absence and 
course failure. These short-term results are promising because chronic absence and course 
failure in grades 9 and 10 are two key predictors that students will not graduate on time 
(Allensworth & Easton, 2005, 2007; Balfanz et al., 2007; Heppen & Therriault, 2008; 
Neild & Balfanz, 2006). However, because the past research linking indicators to on-time 
graduation is correlational, it is not yet known if changing these indicators translates into 
an improvement in on-time graduation rates. Also, EWIMS did not have a detectable 
impact on other measured short-term indicators of risk that are also related to students’ 
likelihood of on-time graduation, including suspensions and low GPAs (although it did 
increase average GPAs). In addition, EWIMS did not have a detectable impact on the 
intermediate outcome of student progress in school (as measured by the number of credits 
students had earned). 

The mechanisms by which EWIMS reduced the percentage of students at risk due to 
chronic absence and course failure are unclear. In particular, it is not known which staff 
actions led to these impacts. The EWIMS theory of action proposes that impacts on 
students may occur as a result of changes in school data use. Although EWIMS imple- 
mentation levels were low overall, the study found a difference between EWIMS and 
control schools in school data culture in the hypothesized direction, favoring EWIMS 
schools. However, the difference was not large enough to be statistically significant. Other 
school-level processes, unmeasured in this study, also may have contributed to impacts on 
students. For example, effects might have emerged for chronic absence and course failure 
if schools prioritized encouraging students to show up and participate in their courses, 
even if they did not have a sophisticated set of interventions. Further research is needed 
to better understand the mechanisms through which EWIMS had an impact on chronic 
absence and course failure. 

Although EWIMS reduced the percentage of students with one or more course failures, 
there was no detectable impact on the related course performance indicator (a GPA of 2.0 
or lower). Sensitivity analyses, however, which used continuous GPA data rather than a 
cutoff, showed a positive impact on average GPA. It is possible that in the first year of adop- 
tion, EWIMS may have had an impact on reducing course failure through modest improve- 
ments in course grades, so that at-risk students may have earned a D instead of an F in some 


Despite low levels 
of implementation, 
the study found that 
EWIMS reduced 
the percentage of 
students who were 
flagged by risk 
indicators related 
to chronic absence 
and course failure; 
however, EWIMS 
did not have a 
detectable impact 
on the intermediate 
outcome of student 
progress in school 
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of their courses, but not necessarily a C or better. Modest improvements in course grades 
would translate into reduced course failures and higher average GPAs, but would not have 
an impact on the percentage of students at risk due to a low GPA (2.0 or lower). 

EWIMS also did not have an impact on the percentage of students who were suspend- 
ed during the study year. This finding may be partially explained by a lack of variability 
and relatively low incidence of reported suspensions in EWIMS and control schools (on 
average, 9 percent in both); by measurement challenges associated with behavioral data, as 
described under Study Limitations below; or by the relative difficulty of intervening with 
students who have more serious disengagement issues with school. 

EWIMS did not have an impact on progress in school (as measured by whether students 
had earned sufficient credits to be on track to graduate within four years), although it did 
reduce course failures. It is unclear why EWIMS did not have an impact on either earning 
insufficient credits or the average number of credits students earned, given that credit 
accrual is based on course performance. 

EWIMS was challenging for the study schools to implement. Initial participation in train- 
ing on the early warning data tool and seven-step process was high among EWIMS schools 
during the summer, but participation declined during the 2014/15 school year. Only two 
schools achieved moderate or high levels of implementation across all seven steps in the 
2014/15 school year. EWIMS schools reported a number of barriers to implementation, 
including challenges related to using the early warning data tool and staffing issues. These 
barriers seemed to be insurmountable for the eight schools that stopped or never began 
implementing EWIMS during the study (22 percent of EWIMS schools). The implementa- 
tion challenges experienced by study schools are important for schools, districts, or states 
to consider when adopting EWIMS or another early warning system. 

Despite low overall levels of implementation, EWIMS schools and control schools adhered 
to their randomly assigned group; many more EWIMS schools than control schools report- 
ed using an early warning system. In addition, more EWIMS schools than control schools 
reported having a dedicated team to identify and support at-risk students. All of the 
remaining treatment contrast analyses — the frequency of course failure data review and 
the number and type of interventions — favored EWIMS schools, with effect sizes typically 
above 0.25, even though the differences were not statistically significant. The one excep- 
tion was that a larger share of control schools than EWIMS schools reported reviewing 
attendance data at least monthly. Impacts on student outcomes might have been greater 
had there been larger differences between EWIMS and control schools in practices for 
identifying and supporting at-risk students. Nevertheless, the study provides an unbiased 
estimate of the impact on study outcomes of EWIMS as it was implemented in the first 
year in 37 schools compared with business-as-usual practices in 36 control schools. 

Future studies should examine whether schools achieve higher overall levels of implemen- 
tation of EWIMS in subsequent years. Future research should also examine whether (and 
how) the impact of EWIMS on chronic absence and course failure in the short term fades, 
grows, or expands in subsequent years to other risk indicators (low GPAs and suspensions), 
to intermediate outcomes (including student persistence in school, student progress in 
school, and school data culture), or to long-term outcomes (including dropout and on-time 
graduation rates). 


Although 
EWIMS had no 
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on the course 
performance 
indicator (a 
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Limitations of the study 


This study has a number of limitations that should be kept in mind. These limitations 
relate to a lack of information on longer term outcomes, the generalizability of the find' 
ings, a lack of detailed information on students in control schools, limited statistical power 
to detect schooUevel effects, and measurement issues. 

• The findings in this report document the early impact of EWIMS, after just one 
year (14 months) of implementation in study schools and do not measure dropout 
or on-time graduation rates over multiple years. Future research should examine if 
and how EWIMS affects dropout and on-time graduation rates and document the 
effects of EWIMS after a longer period of implementation. 

• The findings in this study may not be generalizable to other schools, given that 
the study sample consisted of schools in three REL Midwest Region states that 
were interested in implementing EWIMS and willing to participate in a random 
assignment study. The findings also may not generalize to schools with lower 
on-time graduation rates than the schools that participated in the study, including 
so-called “dropout factories,” defined as schools in which the reported grade 12 
enrollment is 60 percent or less than the grade 9 enrollment three years earlier 
(DePaoli et al., 2015). Three schools that participated in the study had very low 
on-time graduation rates (around 60 percent), but most were higher. The findings 
also may not generalize to other early warning systems, which vary in their com- 
prehensiveness or the degree to which they articulate the implementation process. 

• The indicators used to flag students as at risk of not graduating on time in the 
EWIMS tool were based on prior research and were not locally validated within 
participating districts and schools. It is possible that the results would be different 
with the use of locally validated thresholds and predictors of on-time graduation 
to identify at-risk students. 

• Fourth, the study did not collect detailed information about how control schools 
used data and interventions to support at-risk students. Asking control schools 
to document this information was problematic because tracking this informa- 
tion resembles a key ingredient of EWIMS. For control schools to collect these 
data may have diminished the contrast between EWIMS and control schools 
that was important to providing a fair test of the EWIMS model. Therefore, only 
school-level data in control schools were collected through web-based surveys of 
school leaders. More detailed data from control schools would clarify the contrast 
between the two groups of schools in their early warning indicator-related prac- 
tices during the year of the study. 

• The school-level analyses had limited statistical power; therefore, findings based 
on the survey of participating schools are considered exploratory and should be 
interpreted with caution. 

• Measures of student GPA and progress in school (based on the number of credits 
earned) may have underestimated the impact of EWIMS because for second-year 
students these measures included course failure and credits from the first year of 
high school, prior to full implementation of EWIMS. This limitation does not 
apply to first-year students. 

• The impact of EWIMS on reducing the number of suspensions may not be detect- 
able due to measurement challenges posed by school discipline data. Although 
there is no reason to suspect differences in the quality of school discipline data 
in EWIMS schools and control schools, school discipline records are generally 


Future research 
should document 
the effects of 
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longer period of 
implementation 
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less valid and reliable than other student data because schools may underreport 
behavioral incidents and are inconsistent in reporting suspensions and identifying 
behaviors serious enough to warrant disciplinary action (Irvin, Tobin, Sprague, 
Sugai, & Vincent, 2004; Morrison, Peterson, O’Farrell, & Redding, 2004). 

• Finally, some student outcomes could be affected by teacher awareness of or 
involvement in the EWIMS process. For example, a teacher in an EWIMS school 
could choose to give a student a passing, instead of a failing, grade to keep that 
student off the “flagged” list. In other words, it is possible some of the estimat- 
ed impacts may reflect changes in how teachers react to student behaviors rather 
than to changes in those behaviors. However, this possibility is less plausible for 
attendance data, which more clearly reflect student, not teacher, behavior. Study- 
ing data on other outcomes (such as test scores) could further address this possible 
limitation. 
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Appendix A. Planned implementation of the Early 
Warning Intervention and Monitoring System 


This appendix provides detail about school implementation of the Early Warning Inter- 
vention and Monitoring System (EWIMS). 

Early Warning Intervention and Monitoring System training activities and timeline 

Schools were randomly assigned in March 2014, and initial trainings occurred between 
April 2014 and June 2014. However, full implementation of EWIMS did not begin until 
the 2014/15 school year (table Al). During implementation, EWIMS schools were provided 
access to the EWIMS early warning data tool, print resources, and systematic support from 
a team of technical assistance liaisons. The liaisons were former educators and administra- 
tors, as well as researchers with expertise in dropout prevention strategies. The trainings 
that the technical assistance liaisons provided included: 

• Early warning data tool training. On-site, in-person tool training was offered in spring 
2014- These hands-on training sessions lasted approximately two hours and provid- 
ed guided opportunities for participants to load data into the tool and run reports. 

• Regional trainings. In-person regional training was offered in May and June 2014 to 
all members of each EWIMS school’s team (approximately five to seven members). 
This training provided opportunities to learn about the seven-step EWIMS 
process through a variety of team-based and group activities. The six-hour training 
was hosted regionally in close proximity to two to five EWIMS schools at a time. 

• Early warning data tool refresher training. The on-site, in-person tool training was 
followed by an online refresher training at the beginning of the 2014/15 school 
year (August-October 2014). 

• Site visits. On-site support was offered three times to EWIMS schools during the 
2014/15 school year — typically in the fall, winter, and spring. Site visits included 
brief presentations on EWIMS topics and full EWIMS team meetings guided by 
one of the EWIMS technical assistance liaisons. Additional activities were tai- 
lored to each school’s individual needs. Each site visit lasted two to four hours. 

• WebShares. The EWIMS technical assistance liaisons hosted five WebShares using 
a webinar platform during the 2014/15 school year to provide professional devel- 
opment to EWIMS schools. The WebShares also allowed EWIMS team members 
to share their successes and challenges and to establish communities of practice 
among EWIMS schools. Each of the five WebShares was offered on multiple days 
and times to allow for broad participation. At least one member of the EWIMS 
team at each school was expected to participate. 

• Responsive technical assistance. The liaisons also provided responsive techni- 
cal assistance to the schools throughout the implementation period. As schools 
encountered challenges, they were encouraged to reach out to their assigned 
liaison to request assistance. This support was provided on an as-needed basis. 

Early Warning Intervention and Monitoring System team meetings 

A broad, well-informed school-based EWIMS team was considered by the developers of 
the model to be important for implementation success. According to the model and asso- 
ciated guidance, the EWIMS team may be established as a new team or may build on or 
be integrated into an existing team (for example, a school improvement team, response to 
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Table Al. Timeline of technical training and implementation schedule for Early Warning Intervention 
and Monitoring System schools during the 2013/14 and 2014/15 school years 


2014 

2015 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Technical training schedule 

Early warning data tool 
training 


Refresher early warning data 
tool training 










Regional training 
on the seven steps 

















Site visit 1 



Site visit 2 

Site visit 3 








Web- 

Share 1 

Web- 

Share 2 


Web- 

Share 3 



Web- 

Share 4 

Web- 

Share 5 


School implementation schedule 


Initial import of student data into early warning data tool for 
first- and second-year students and flagging of at-risk students® 













Import student background 

characteristics and 

attendance data after the 

first 20 or 30 days and flag 
at-risk students; and import 
all student data after the first 

grading period for first- and 
second-year students and 
flag at-risk students 11 


Import all student 

data after the 

second grading 
period for first- 
and second-year 
students and flag 

at-risk students 


Import all student 

data after the 

third grading 
period for first- 
and second-year 
students and flag 

at-risk students 

Import all student 

data after the 

fourth grading 
period for first- 
and second-year 
students and flag 

at-risk students 





Assign students to interventions, monitor progress, and revise intervention assignments (ongoing throughout the 
school year) 





Initial team 

meeting 0 

Team 

meeting 

Team 

meeting 

Team 

meeting 11 

Team 

meeting 

Team 

meeting 

Team 

meeting 

Final team 

meeting 


EWIMS is the Early Warning Intervention and Monitoring System. 

a. The initial data import included uploading student demographics, a course catalog, and a list of all interventions available to stu- 
dents. If available, schools imported incoming risk data, including data from grade 8 for first-year students and data from the previous 
school year for second-year students. 

b. The grading periods in the timeline are 9-week quarter grading periods. Schools using a trimester grading period schedule imported 
data in the fall (around November), winter (around February), and spring (around May). 

c. EWIMS teams were expected to meet monthly to review students' risk status, assign students to interventions, and monitor stu- 
dents' progress in interventions. 

d. Many schools held only one EWIMS team meeting during December and January, but then met more frequently throughout the rest of 
the year. 

Source: Authors' compilation. 


intervention team, or student support team). An existing team that takes on the resporv 
sibility to use the EWIMS tool for dropout prevention efforts should include a broad 
representation of staff within the school and, ideally, the administration (for example, 
principals, teachers, district administrators, specialists). Ideally, EWIMS teams also will 
have district and feeder school representation. 

In this study, the EWIMS technical assistance liaisons asked each treatment school to 
form an EWIMS schoohbased team in April 2014 that consisted of an EWIMS team 
chair, a primary user of the early warning data tool, and three to five other members who 
had knowledge of individual students at the school and of the interventions available to 
students. The chair was responsible for organizing the EWIMS team, handling commu- 
nications with the project’s implementation team, and ensuring team members followed 
through on tasks. The primary user of the early warning data was responsible for preparing 
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and importing data into the early warning data tool, providing other team members access 
to the tool by managing user accounts, and ensuring the data and reports were available. 
Other team members were responsible for attending team meetings and following through 
on action items assigned by the team. The roles described could have been filled by one or 
more individuals at their discretion. Each team was responsible for fulfilling these roles in 
some way. 

The EWIMS schoohbased team was expected to conduct the bulk of its work in the 
context of monthly meetings. It was expected that teams would meet at least once per 
month throughout the 2014/15 school year, with approximately 10 meetings per school. 
An agenda for each upcoming meeting was to be prepared at the end of the prior meeting, 
with standing agenda items including reviews of the data from the early warning data tool, 
actions taken for individual or groups of students, previous meetings’ action items (ongoing 
or completed), new action items, and communication with staff and leadership. Notes were 
to be taken at each meeting and were to include action items assigned to specified individ- 
uals to accomplish. Agenda, meeting notes, and a faculty/staff sign-in sheet were be kept 
on file to provide a record of the team’s work. 

Importing data into the early warning data tool, and documenting students’ assignment to interventions 

A member of the EWIMS team was responsible for importing data into the tool to facili- 
tate the EWIMS team’s use of the early warning indicator data. Participating schools were 
expected to upload attendance data after the first 20 or 30 days of school and after every 
grading period. Course performance, grade point averages, and behavioral data were also 
expected to be uploaded after every grading period. 

The EWIMS team member responsible for importing student data also was expected to 
import a list of interventions available in the school. EWIMS teams then were expected 
to use features within the tool to document their assignment of students to interventions, 
including dates. The EWIMS technical assistance liaisons encouraged EWIMS teams to 
match students with interventions and to monitor students’ data and their at-risk status 
on the early warning indicators to inform ongoing decisions about the match between 
students and interventions. 

Use of incoming risk flags during the 2014/15 school year 

Use of the incoming risk flag in the EWIMS tool was not a major focus of implementation. 
The incoming risk flag was locally defined by schools using information from the prior 
year. The flag was not an automatic flag triggered by any particular data element in the 
tool (like those for chronic absence or course failure) but was imported by schools as part of 
the student demographic information. The EWIMS technical assistance liaisons advised 
schools to flag rising second-year students based on the off-track flag using data from their 
first year and to flag rising first-year students based on grade 8 data on attendance, course 
performance, behavior, or a list from middle school counselors. Data culled from the early 
warning data tool suggest that 23 of the EWIMS schools had at least some data in the 
incoming risk flag field. Among these 23 schools, first-year students more consistently had 
incoming risk flag data (45 percent) than did second-year students (38 percent). However, 
it is not clear what information each school used to determine incoming risk. 
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Appendix B. Recruitment, random assignment, and study sample 


This appendix provides information about the study sample, including the recruitment 
and selection of participating schools, random assignment, school attrition, the analytic 
samples, and baseline equivalence. 

Recruiting and selecting schools to participate in the study 

The study aimed to recruit 72 high schools to provide sufficient statistical power to 
examine the impact of the Early Warning Intervention and Monitoring System (EWIMS) 
on student outcomes. Recruitment focused on three Midwest states that had an explicit 
focus on improving on-time graduation rates. None of the three states were implementing 
a statewide early warning system similar to EWIMS at the time of recruitment. 

School eligibility criteria. Schools were eligible for the study if they had at least 130 grade 
9 students, had an on-time graduation rate between 25 percent and 95 percent, were not 
using an early warning system, and were not using data in ways that closely mirrored the 
EWIMS seven-step process. Eligibility screening during recruitment focused on schools’ 
existing practices related to the use of attendance and course performance data for iden- 
tifying students who may be at risk of dropping out of high school, using these data to 
identify the root causes of chronic absences and course failure, and relying on these data 
to assign students to interventions to get students back on track. Attendance and course 
performance data were used in the eligibility screening because they were considered to be 
the most commonly available early warning indicators identified in prior related work. The 
screening did not focus on discipline or other types of data. 

To operationalize schools’ baseline data-driven dropout prevention efforts, each school was 
rated on a 6-point scale based on information gathered during a school screening interview 
and recruitment site visits. Definitions of schools’ baseline data-driven dropout prevention 
efforts were as follows: 

• 1 = very low data-driven dropout prevention efforts. Schools did not flag students 
as at risk of not graduating on time or assign students to interventions on the basis 
of course grades or attendance data in grades 9 and 10. 

• 2 = low data-driven dropout prevention efforts. Schools flagged students as at risk 
of not graduating on time and assigned students to interventions on the basis of 
attendance but not course failure data in grades 9 and 10. 

• 3 = moderate data-driven dropout prevention efforts. Schools flagged students as 
at risk of not graduating on time and assigned students to interventions on the 
basis of course failure but not attendance data in grades 9 and 10. 

• 4 = high data-driven dropout prevention efforts. Schools flagged students as at risk of 
not graduating on time and assigned students to interventions on the basis of atten- 
dance and course failure data in grades 9 and 10, but were making little to no use of 
other data sources (for example, state test scores or formative assessments) to identify 
individual student learning needs and to understand root causes of risk indicators. 

• 5 = very high data-driven dropout prevention efforts. Schools flagged students as at 
risk of not graduating on time and assigned students to interventions on the basis 
of attendance and course failure data and used other data sources (for example, 
state test scores and formative assessments) at the individual level to understand 
student-specific learning needs. 
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• 6 = using an early warning system. Schools were already using an early warning 

system to inform how they flagged students as at risk of not graduating on time 
and assigned students to interventions. 

Schools with a score of 5 or 6 were deemed ineligible to participate; schools with a score of 
4 or lower were considered eligible. Treatment and control schools were similarly distribut' 
ed across the categories of baseline dropout prevention efforts; see the “Student and school 
baseline equivalence” section and table B5 later in this appendix for more detail. 

Recruitment process. Recruitment took place during the 2013/14 school year. During 
recruitment, schools were first identified as potentially eligible based on size and on-time 
graduation rates using extant data. The study team contacted 688 potentially eligible 
schools (142 in State 1, 225 in State 2, and 321 in State 3) by email and follow-up phone 
call to provide an introduction to the project (table Bl). Following this initial communi- 
cation, 282 schools responded to emails and phone calls. Interested schools were invited 
to participate in a telephone screening interview. The purpose of the telephone screening 
interview was to learn more about the schools’ use of data to identify at-risk students, their 
dropout prevention strategies, and any other initiatives that might interfere with their par- 
ticipation in EWIMS. During the telephone screening interview, the recruitment team 
also confirmed that the school’s structure, size, and on-time graduation rate met the study’s 
eligibility criteria. Of the 282 interested schools, 130 participated in a telephone screening 
interview. Of those 130 schools, 95 participated in site visits. The purpose of the site visit 
was to present the study and describe early warning systems generally and the EWIMS 
process, in particular, to school staff and to confirm that EWIMS would not duplicate 
school practices that existed at the time. 


Table Bl. Recruitment samples from the 2013/14 school year 

Recruitment phase 

Number of 
schools 

Potentially eligible schools 

688 

School did not respond after first contact 

406 

Schools that responded after first contact (phone or email) 

282 

School did not respond after second contact 

95 

Recruitment team could not schedule screening interview 

57 

Schools that participated in the screening interview 

130 a 

School declined to participate in the study 

24 

Entire school district declined to participate 

2 

Communication between recruitment team and school ceased 

4 

Recruitment team deemed the school ineligible for the study 

27 

School’s on-time graduation rate exceeded eligibility criteria 

4 

School’s processes too closely mirrored EWIMS (scored a 5 or 6 on the screening tool) 

21 

School structure (first- or second-year students housed on separate campuses) 

1 

School could not adhere to random assignment 

1 

Final recruited school sample 

73 


a. Of the 130 screened schools, 95 also received an in-person site visit. All 73 schools that participated in 
the study had a site visit. 

Source: Authors' compilation. 
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Schools that remained interested and eligible. Seventy-three schools received and signed 
formal memoranda of understanding to participate in the project (see table Bl). Although 
there was a target of 72 schools, 73 were included because the marginal cost of data col- 
lection for the last school was relatively low. This school was the third school in a dis- 
trict with three schools, and the associated data collection costs were low, given that data 
already was being collected from this district (table B2). 

Description of the 73 recruited and participating schools. The 73 schools that signed 
formal memoranda of understanding to participate in the project varied in school size, 
on-time graduation rate, and baseline data-driven dropout prevention efforts. Schools were 
targeted for recruitment if they enrolled at least 130 grade 9 students. The majority of 
schools that joined the study had between 160 and 400 grade 9 students, and approxi- 
mately a third of the schools enrolled more than 500 grade 9 students. Although schools 
with graduation rates between 25 percent and 95 percent were eligible to participate in 
the study, study schools typically had relatively higher graduation rates. In 2012/13, only 
3 of the schools had graduation rates below 60 percent and only 6 had graduation rates 
below 75 percent; the majority of the schools had on-time graduation rates of 88 percent 
or higher. 

Schools were eligible to participate in the study if they received a score below 5 on a 
6-point scale in a screening protocol designed to capture data-driven dropout prevention 
efforts (the scores for this scale are described earlier in this appendix). The majority of 
participating schools had a score of 3 or lower on this 6-point scale. 12 

Random assignment 

The 73 schools that signed a memorandum of understanding were randomly assigned to 
implement EWIMS (EWIMS schools) or continue their usual practices for identifying and 
supporting students at risk of not graduating on time (control schools). Random assign- 
ment was conducted using matched pairs and cluster randomization. Schools were matched 
using a Mahalanobis distance metric that describes how different any two schools are 
based on their graduation rates, school size, and baseline data-driven dropout prevention 
efforts (school eligibility score). The Mahalanobis metric is based on a measure of overall 
similarity (“Mahalanobis distance”) between two units with respect to a pair of covariates. 
Mahalanobis distance is calculated on the basis of covariate differences between the units 
(for example, schools) and the sample variance-covariance matrix (Cochran & Rubin, 


Table B2. Number of districts that had one or more eligible schools and the number 

of schools included in those districts 



Number of study 
schools in the 
same district 

Number of districts 

Number of schools 

Number of matched pairs 
for random assignment 

1 

58 

58 

29 

2 

4 

8 

4 

3 

1 

3 

1 

4 

1 

4 

2 

Total 

64 

73 

36 

Source: Authors' compilation. 
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1973; Rosenbaum & Rubin, 1985; Rubin, 1980)' Mahalanobis matching is appropriate in 
situations like this one, in which each school could potentially be matched with any other 
school in the pair. 

Once all of the Mahalanobis distances had been calculated, the matched pairs were con- 
structed using optimal matching, which minimized the total of the within-pair distances 
across all possible pairings. The optimal matching was conducted using a web application 
that implements the nbpMatching (Nonbipartite Matching) package in R written by Lu, 
Greevy, and Beck (2007). Because of the importance of the local context around dropout 
prevention (for example, state dropout prevention policies or district dropout prevention 
policies and practices), the matching procedure required an exact match on state (for 
example, a school in State 1 could be paired only with another school in State 1 and not 
with a school in State 2) and district in cases where more than one school in a district was 
recruited. The decision to force matches within district affected 7 of the 36 matched pairs 
(see table B2). 

Next, one school within each matched pair was randomly assigned to the EWIMS group 
and the other to the control group. The only exception to this process was in one dis- 
trict with three recruited schools. In this district, the two schools that were most similar 
were matched with each other to create a pair and then each was randomly assigned to 
either the EWIMS group or the control group. The remaining school was then randomly 
assigned to either implement EWIMS or to the control group by a “coin flip.” Thus, all 
schools in the study had a 50 percent chance of being assigned to EWIMS or the control 
group. Thirty seven schools were randomly assigned to implement EWIMS and 36 were 
randomly assigned to the control group. 

School attrition 

Schools were considered to have attrited from the study only if they no longer participated 
in studentdevel data collection or if their matched pair counterpart did not participate in 
data collection. 13 In total, four EWIMS schools (and their matched pairs) formally with- 
drew from the study: these included two schools that had stopped implementing EWIMS 
and two EWIMS schools that implemented EWIMS but did not provide extant student 
data and were considered to have withdrawn. When a school officially withdrew from the 
study, the study team discontinued data collection for that school and its matched coun- 
terpart in the control group. Therefore, the four control schools that were matched to the 
four EWIMS schools that withdrew were also considered to have withdrawn, reducing the 
analytic sample from 73 schools to 65 schools (33 EWIMS schools, 32 control schools). 

School and student analytic samples 

School analytic sample. Because of attrition, the school analytic sample decreased from 
73 to 65 schools (33 EWIMS and 32 control schools) for studentdevel analyses. The school 
analytic sample size also varied depending on levels of missing data for each outcome vari- 
able. For example, all 65 schools in the sample provided data on chronic absence and 
course failure, but only 63 schools provided data for suspensions and progress (credits 
earned) and only 57 schools provided grade point average (GPA) data. Thus, the school 
analytic samples differed by the outcome measure used in each model (figure Bl). 
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Figure Bl. School and student sample sizes from recruitment to analytic samples 



EWIMS is the Early Warning Intervention and Monitoring System. GPA is grade point average. 

Note: Four EWIMS schools could not be included in the sample because student outcome data were not 
provided. Of the four EWIMS schools, two stopped implementing EWIMS and withdrew from the study and two 
implemented EWIMS but did not provide student outcome data. The four control schools matched to these 
schools were also not included in the analytic sample. The student n represents the number of students pres- 
ent in the schools in the sample. The analysis n represents the number of students with nonmissing values 
on the outcome measure. Because students were missing values for each of the outcomes, the analysis n is 
smaller than the student n for each outcome. 

Source: Authors' compilation. 


For schooUevel analyses that relied on self-reported survey data (those that measured the 
treatment contrast or the impact of EWIMS on school data culture), schools were included 
in the sample if they responded to the survey. A total of 67 schools (33 EWIMS schools 
and 34 control schools) replied to at least one item on the school leader survey. 

Student analytic sample. A total of 37,671 first- and second-year students in 65 schools 
provided at least one data element: 18,634 students in EWIMS schools and 19,037 students 
in control schools were included in the study (table B3). 

The student analytic sample was reduced further due to missing student-level data. The 
student analytic samples ranged from 30,086 to 35,888 across outcomes (figure Bl). The 
school and student sample sizes from recruitment and random assignment in March 2014 
through data collection at the end of the 2014/15 school year are presented in figure 
Bl. The samples for each analysis show the number of pairs, the number of schools, the 
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Table B3. Number of first- and second-year students and total sample size, by 


treatment group 

1 Grade 

Overall 

EWIMS schools 

Control schools 1 

First-year students 

18,748 

9,217 

9,531 

Second-year students 

18,923 

9,417 

9,506 

Total 

37,671 

18,634 

19,037 


EWIMS is the Early Warning Intervention and Monitoring System. GPA is grade point average. 

Note: This study followed two groups of students (first-year students and second-year students) in EWIMS 
schools and control schools. First-year students were enrolled in grade 9 in the 2014/15 school year; second- 
year students were enrolled in grade 9 in the previous (2013/14) school year and enrolled in either grade 9 or 
10 in the 2014/15 school year. 

Source: Authors' compilation. 


number of students in each sample, and the number of students included in each analysis 
who were not missing outcome data. 

Rates of missing data at the school and student levels 

The extent of missing data at the school level varied across the different outcomes used 
in the primary impact models. Also, if a school failed to provide studentdevel data for an 
outcome measure, both it and its matched counterpart were dropped from the student- 
level analyses. As a result, there was no differential attrition between EWIMS and control 
schools at the school level due to missing data on outcome measures in the analytic 
samples. As a test of the robustness of the results, a sensitivity analysis was run that used a 
common analytic sample for all outcomes (see tables D1 and D2 in appendix D). 

Within these analytic samples for each outcome, the percentage missing was calculated 
using a denominator equal to the number of students in the matched pairs of schools that 
provided data for each outcome. Data were missing for between 3 percent and 8 percent of 
students in the analytic samples, and there were no statistically significant differences in 
the level of missingness by condition (table B4). 

Because the rates of missing data at the student level were low, the primary impact models 
did not use any missing data strategy to correct for missingness on the outcomes. As a 
sensitivity analysis, results were also estimated for models that adjusted for missingness on 
outcomes using inverse probability weighting. With this approach, students who were not 
missing the outcome measure were weighted to represent students who were (Puma, Olsen, 
Bell, & Price, 2009). These inverse probability weights were estimated using a model that 
contained the following covariates: state of residence, grade, whether the student was in 
an EWIMS school, all demographic variables, grade 8 test scores, and dummy variables 
representing each matched pair. The results are similar to those produced with the primary 
impact model specifications (see table D1 in appendix D). 

Across the analytic samples, the percentage of students missing demographic covariates 
or the grade 8 state standardized test scores ranged from less than 1 percent to 21 percent 
for EWIMS schools and less than 1 percent to 18 percent for control schools. Age, gender, 
and race/ethnicity data were missing for less than 1 percent of students in EWIMS schools 
and control schools across each analytic sample. Data on English learner status and special 
education status (as indicated by having an Individualized Education Program) at baseline 
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Table B4. Number and percentage of students missing data for each outcome, by treatment group 


Outcome 


Overall 


EWIMS schools 

Control schools 

Standard- 

p-value 

Analysis 
(number of 
students) 

Missing 
(number of 
students) 

Percent 

missing 

Analysis Missing 
(number of (number of 
students) students) 

Percent 

missing 

Analysis Missing 
(number of (number of 
students) students) 

Percent 

missing 

ized 

difference 

(Hedges 

g) 

Chronic absence 

35,888 

1,783 

5 

17,411 

1,223 

7 

18,477 

560 

3 

0.17 

0.144 

Course failure 

35,133 

2,538 

7 

17,410 

1,224 

7 

17,723 

1,314 

7 

-0.01 

0.934 

Low GPA 

30,086 

2,336 

7 

15,253 

1,005 

6 

14,833 

1,331 

8 

-0.08 

0.700 

Suspended 

35,558 

1,068 

3 

17,557 

541 

3 

18,001 

527 

3 

0.01 

0.285 

Progress in schoo 

1 35,050 

1,857 

5 

17,499 

743 

4 

17,551 

1,114 

6 

-0.09 

0.855 


EWIMS is the Early Warning Intervention and Monitoring System. GPA is grade point average. 

Note: Using a multilevel logistic regression model, the binary indicator of whether the outcome was missing was regressed on the treat- 
ment indicator that identified schools as EWIMS or control schools and a set of dummy variables that captured school matched pairs. 
The p-value was associated with the test of whether there was a statically significant difference in missing data by condition for each 
outcome. 

Source: Authors' analysis based on extant student records from schools, school districts, and state education agencies described in 
appendix C. 


were missing for 18 percent to 21 percent of students in EWIMS schools and for 15 percent 
to 18 percent of students in control schools. Data on eligibility for the federal school lunch 
program at baseline were missing for 22 percent to 26 percent of students in EWIMS 
schools and for 17 percent to 20 percent of students in control schools. Schools had a 
difficult time providing these data for the prior school year or before random assignment, 
especially for students who were in grade 9 in 2014/15. Across the analytic samples, grade 
8 test scores in reading and math were missing for 18 percent to 21 percent of students in 
EWIMS schools and for 12 percent to 15 percent of students in control schools. 

There was no differential missingness by condition for gender, racial/ethnic minority, 
and age data. However, there were statistically significant differences in the proportion 
of students with missing covariate data in EWIMS and control schools on state math and 
reading test scores and on baseline English learner status and special education status in 
certain analytic samples. 

• Rates of missing data on grade 8 state math and reading test scores were higher in 
EWIMS schools than in control schools for some analytic samples, including the 
samples used to evaluate the impact of EWIMS on chronic absence, course failure, 
and grade point average. 

• Rates of missing data on baseline English learner status and special education 
status were higher in EWIMS schools than in control schools for some analytic 
samples, including the samples used to evaluate the impact of EWIMS on all stu- 
dennlevel outcomes except progress in school. 

The differential missingness was addressed with the use of missing data flags and mean 
imputation for missing covariates in the analyses. Specifically, a dummy variable adjust' 
ment was used to retain in the primary impact models students who were missing data 
on one of the baseline covariates. Missing cases were set to a constant (the mean), and 
dummy indicators for missing data were included in the model. This strategy for handling 
missing covariate data has been shown to produce impact estimates and standard errors 
with low bias (Puma et al., 2009). As an additional check of the robustness of the results, 
models were also estimated without covariates (see table D1 in appendix D). 


B-7 




Student and school baseline equivalence 


To determine if random assignment produced two groups of similar schools, baseline chan 
acteristics were analyzed for all schools and students in the intent-to-treat study sample 
and analytic samples. Baseline school characteristics included grade 9 enrollment and 
total school enrollment, on-time graduation rates, whether the school was categorized as 
moderate or high on baseline data-driven dropout prevention efforts as assessed during 
recruitment, the percentage of students eligible for the federal school lunch program, and 
the percentage of racial/ethnic minority students. 

Baseline student characteristics included grade 8 standardized state test scores in reading 
and math, gender, racial/ethnic minority status, English learner status, special education 
status, and eligibility for the federal school lunch program. 

In the intent-to-treat sample of all 73 schools, baseline equivalence was achieved across all 
school-level baseline measures (table B5). EWIMS and control schools had similar base- 
line school size (both grade 9 and total enrollment), on-time graduation rates, data-driven 
dropout prevention efforts, and student characteristics. EWIMS and control schools also 
served student populations with similar demographic characteristics and prior achieve- 
ment on state assessments in reading and math (see table B5). 

Baseline equivalence within each analytic sample was also examined. Baseline charac- 
teristics by condition were not significantly different for any of the analytic samples, and 
all standardized differences were 0.20 or less, with one exception. In the analytic sample 
for GPA, EWIMS schools had an average of 41 percent of students eligible for the federal 
school lunch program, control schools had an average of 45 percent, a standardized differ- 
ence of -0.22 that was not statistically significant at the p < 0.05 level. 

The study team examined the baseline equivalence by grade. Contrasts of the 13 baseline 
characteristics by condition yielded no statistically significant differences at the p < 0.05 
level for the first-year student or second-year student samples. All standardized differences 
of the baseline characteristics within grade were less than 0.20. 
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Table B5. Baseline characteristics of schools and students in the randomly assigned sample, overall and by condition prior to random 
assignment in March 2014 




Overall 



EWIMS schools 



Control schools 


Stan- 


Characteristic 

School 

n 

Student 

n 

Mean 

Standard 

deviation 

School 

n 

Student 

n 

Mean 

Standard 

deviation 

School 

n 

Student 

n 

Mean 

Standard dardized 
deviation difference 

p-value 

School characteristics 

Grade 9 school enrollment 

73 

na 

321.34 

142.84 

37 

na 

318.97 

137.54 

36 

na 

323.78 

150.01 

-0.03 

0.687 

Total school enrollment 

73 

na 

1,213.52 

546.22 

37 

na 

1,165.03 

523.96 

36 

na 

1,263.36 

571.25 

-0.18 

0.156 

On-time graduation rate (%) 

73 

na 

86.92 

7.93 

37 

na 

86.72 

9.03 

36 

na 

87.13 

6.74 

-0.05 

0.839 

High baseline data-driven 
dropout prevention efforts (%) a 

73 

na 

61.64 

48.96 

37 

na 

59.46 

49.77 

36 

na 

63.89 

48.71 

-0.09 

0.567 

Eligible for the federal school 
lunch program (%) 

73 

na 

42.79 

18.63 

37 

na 

41.18 

18.53 

36 

na 

44.45 

18.84 

-0.17 

0.239 

Racial/ethnic minority 
enrollment (%) 

73 

na 

24.03 

26.24 

37 

na 

23.80 

26.88 

36 

na 

24.26 

25.94 

-0.02 

0.689 

Student demographic characteristics 

Female (%) 

65 

37,541 

48.85 

49.99 

33 

18,568 

49.13 

49.99 

32 

18,973 

48.57 

49.98 

0.01 

0.284 

Racial/ethnic minority status (%) 

65 

37,545 

24.67 

43.11 

33 

18,564 

26.20 

43.97 

32 

18,981 

23.18 

42.20 

0.07 

0.906 

Students in special education (%) 

65 

30,482 

13.29 

33.94 

33 

14,760 

13.30 

33.96 

32 

15,722 

13.27 

33.93 

0.00 

0.932 

English learner students (%) 

65 

30,481 

2.99 

17.04 

33 

14,760 

3.16 

17.49 

32 

15,721 

2.84 

16.60 

0.02 

0.288 

Eligible for the federal school 
lunch program (%) 

62 

29,711 

43.77 

49.61 

31 

14,213 

45.59 

49.81 

31 

15,498 

42.11 

49.37 

0.07 

0.921 

Student prior achievement 

Grade 8 state standardized test 
scores in reading 

62 

31,165 

612.01 

161.80 

30 

14,903 

615.43 

161.21 

32 

16,262 

608.87 

162.27 

-0.05 

0.871 

Grade 8 state standardized test 

scores in math 

62 

31,196 

628.34 

152.59 

30 

14,903 

632.52 

151.13 

32 

16,293 

624.52 

153.81 

-0.04 

0.488 


EWIMS is the Early Warning Intervention and Monitoring System, n is the number of schools or students, na is not applicable because these are characteristics at the school level. 

Note: None of the differences between groups of schools or groups of students were statistically significant at the p < .05 level. Hedges’ g was used to compute the standardized 
difference. A linear regression model with a treatment indicator that identifies schools as EWIMS schools or control schools and a set of dummy variables that captures school 
matched pairs was used to test the school characteristic differences. A multilevel linear regression model with students at level 1 and schools at level 2, a treatment indicator, and 
a set of dummy variables that captures school matched pairs was used to test the student characteristic differences. 

a. This measure is the percentage of schools scoring 3 or 4 (moderate or high) on the scale, while omitting those schools scoring 5 or 6 (very high), since they were not included in 
the study. 

Source: Authors’ analysis based on school characteristics from 2012/13 and extant student records from schools, school districts, and state education agencies described in ap- 
pendix C. Extant student records from the 2012/13 school year are used for the second-year students, and data from the 2013/14 school year are used for the first-year students. 






Appendix C. Data collection and analytic methods 


This appendix provides detail about the methods used in the study, including the data 

sources, data collection, and analytic methods for each research question. 

Description of data 

The following types of data were collected: 

• Extant student records from schools, school districts, and state education agencies 
for students in schools implementing the Early Warning Intervention and Moni- 
toring System (EWIMS) and control schools between the 2012/13 school year and 
the end of the 2014/15 school year. These data were used for student background 
characteristics and to operationalize student outcomes. 

• School characteristics of participating schools that are publicly available from the 
Common Core of Data (CCD) and state education agencies (U.S. Department 
of Education, 2015). These data were used for random assignment and baseline 
equivalence. 

• School leader responses on a web-based survey, administered in spring 2015, to 
measure school data culture, collect information about the number and type of 
interventions used to support at-risk students in EWIMS and control schools, and 
operationalize the treatment contrast. The survey was also administered in spring 
2014, and data from this administration were used only as an additional data 
source for identifying interventions offered in EWIMS schools. 

• Extant documents on EWIMS implementation collected between spring 2014 and 
spring 2015 by the EWIMS technical assistance liaisons for EWIMS schools only 
(for example, attendance sheets and school implementation summaries). 

• Monthly logs recording the content and frequency of EWIMS team meetings 
during the 2014/15 school year in EWIMS schools only. 

• Reports from early warning data tools that captured tool use through spring 2015 
in EWIMS schools only. 

• Interviews with EWIMS team members administered in spring 2015 in EWIMS 
schools only. 

Data are displayed by research question in table Cl. 
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Table Cl. Data from the 2012/13, 2013/14, and 2014/15 school year used to address each research question 

1 Research questions 

Data elements 

Data sources 

Data time frame 1 

Impact 1: What is the impact of EWIMS 

on indicators of student risk? 

• Student demographic characteristics 

■ Grade 8 test scores (prior achievement) 

• Chronic absence (total number of absences 
for the school year) 

• Course failure (course titles and course 
grades) 

■ Grade point average 

• Suspension (suspensions and expulsions) 

State education agencies, 
school districts, schools 

2012/13 and 2013/14 for prior achievement; 
2014/15 

Impact 2: What is the impact of EWIMS 
on student progress in school? 

■ Student demographic characteristics 

■ Grade 8 test scores (prior achievement) 

• Credits earned 

State education agencies, 
school districts, schools 

2012/13 and 2013/14 for prior achievement; 
2014/15 

Exploratory 1: What is the impact of 

EWIMS on school data culture? 

• School data culture (context for data use, 
concrete supports for data use, data-driven 
student support, and barriers to data use) 

School leader survey 

Spring 2014/15 

Implementation 1: To what extent 
did EWIMS schools participate in the 
professional development provided 
and implement the EWIMS seven-step 
process? 

■ Participation in EWIMS training 

• EWIMS training satisfaction surveys 

• Level of implementation of the seven-step 
EWIMS process 

School leader survey, EWIMS 
monthly logs, early warning 
data tool reports, EWIMS team 
interview, extant documents 
on EWIMS implementation 

April 2014-June 2015 

Implementation 2: What barriers to 
implementation did EWIMS schools 
experience? 

Barriers 

School leader survey, EWIMS 
monthly logs, early warning 
data tool reports, EWIMS team 
interview (and exit interviews), 

extant documents on EWIMS 

implementation 

April 2014-June 2015 

Implementation 3: What types of 
interventions did EWIMS schools provide 
to students identified as at risk, and what 
percentage of students received those 
services? 

Student assignment to interventions 

School leader survey, early 
warning data tool reports, 

EWIMS team interview 

April 2014-June 2015 

Implementation 4: To what extent did 

EWIMS and control schools differ in their 

practices for identifying and supporting 
students at risk of not graduating on 

time? 

Frequency of data review on attendance and 
course performance, interventions available 
to support students, the presence of an early 
warning system and dedicated team to identify 
and support at-risk students 

School leader survey 

June 2015 

EWIMS is the Early Warning Intervention and Monitoring System. 

Source: Authors' compilation. 




Data were requested for all participating schools. Of the 73 schools that were random- 
ly assigned to the EWIMS and control group samples, the school-level data collection 
rates of the individual data elements ranged from 78 percent to 95 percent: 78 percent 
to 95 percent for EWIMS schools and 86 percent to 94 percent for control schools (table 
C2). School-level data sources with student-level data could have missing student data. 
Student-level missing data are detailed earlier as part of the description of samples and 
attrition (see appendix B). 

Extant student records. All student-level data were collected from extant records from 
state education agencies, districts, and the schools themselves, depending on the data 
element; there was no student-level primary data collection burden on schools. For the 
second-year students (students in grade 9 in 2013/14), baseline data were collected from the 
2012/13 school year and for the first-year students (students in grade 9 in 2014/15), baseline 
data were collected from the 2013/14 school year. Student outcome data were collected for 
the 2014/15 school year for first- and second-year students. The data collection rates for the 
individual data elements ranged from 78 percent to 89 percent for EWIMS schools and 
86 percent to 89 percent for control schools (see table C2). Of the 73 schools randomly 
assigned to the EWIMS and control group samples, 75 percent provided all of the extant 


Table C2. School-level data collection rates, by condition and overall, during the 


2014/15 school year 

Data collected 

EWIMS 
schools 
(n = 37) 
Percent 

Control 
schools 
(n = 36) 
Percent 

All 

schools 
(n = 73) 
Percent 

School characteristics 

School leader survey (spring 2015) 

89 

94 

92 

Implementation summaries from EWIMS school liaisons 

78 

na 

na 

EWIMS training satisfaction surveys 3 

95 

na 

na 

EWIMS monthly logs 

89 

na 

na 

Early warning data tool reports 

84 

na 

na 

EWIMS team interviews (and exit interviews) 

95 

na 

na 

Extant student records 

Grade level 

89 

89 

89 

Demographic characteristics 

84 

89 

86 

Grade 8 test scores 

81 

89 

85 

Attendance 

89 

89 

89 

Course failure 

89 

89 

89 

Grade point average 

78 

86 

82 

Suspension 

86 

89 

88 

Credits earned 

86 

89 

88 

Enrollment and exit codes 

84 

89 

86 


na is not applicable because these data were not collected from control schools. EWIMS is the Early Warning 
Intervention and Monitoring System, n is the number of schools or students. 

Note: Only two schools are missing demographic records, and records are missing only eligibility for the fed- 
eral school lunch program. 

a. Satisfaction surveys were collected after each of the 11 EWIMS trainings, and response rates ranged from 
60 percent to 95 percent. Most satisfaction survey response rates were above 80 percent (for 9 of the 11 
trainings); the two exceptions were site visit 1 (78 percent) and site visit 2 (60 percent). 

Source: Authors' compilation. 
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student records, including grade level, demographic characteristics, grade 8 test scores, 
attendance, course failure, grade point averages (GPA), number of suspensions, credits 
earned, and enrollment and exit codes. Seventy-seven percent of the 73 schools provided 
all of the extant student records to operationalize the outcome measures (that is, chronic 
absence, course failure, low GPA, number of suspensions, and progress in school). 

The data elements collected included the following: 

• Demographic characteristics. Measures included date of birth, gender, race/ethnicity, 
special education status (as indicated by having an Individualized Education 
Program), English learner status, eligibility for the federal school lunch program, 
and grade level. Age was calculated as of September 1, 2013, using the date of 
birth variable. Race/ethnicity was coded into binary indicators denoting whether 
students represented racial/ethnic minority groups. Gender, special education 
status, English learner status, and eligibility for the federal school lunch program 
were coded as binary indicators. Grade level in the 2013/14 and 2014/15 school 
years was used as an indicator of the grade level for each student. 

• Grade 8 state standardized tests. Grade 8 reading and math scores from the 
2012/13 and 2013/14 school years served as measures of prior achievement for the 
second-year students and first-year students, respectively, in the primary impact 
models. Grade 8 test scores in reading and math were standardized using the mean 
and standard deviation for the sample within each state and grade (^-scores). 

• Attendance. Attendance data included the total number of absences and total days 
enrolled in the 2014/15 school year. Total number of absences included excused 
and unexcused absences. Absences and enrollment counts were provided by days, 
hours, or class periods, depending on how attendance was reported for the school 
year by the school, district, or state education agency. 

• Course failure. Course data included course name and course grade for all courses 
attempted for the first' and second-year students in the study schools during the 
2014/15 school year. 

• Grade point averages. GPA data included cumulative high school GPA through the 
2014/15 school year for the first- and second-year students in study schools. 

• Suspensions. Suspension data included total number of suspensions or total days 
suspended in the 2014/15 school year. For some schools, 22 in total, it was not clear 
which was provided. “Total suspensions” combines in-school and out-of-school 
suspensions. 

• Credits earned. Credits data included the cumulative number of high school credits 
earned through the 2014/15 school year for the first- and second-year students in 
study schools. 

School characteristics. The following school-level extant data were collected for the 73 
study schools to inform random assignment and to assess baseline equivalence. Grade 9 
enrollment records and on-time graduation rates were used to create the matched pairs for 
random assignment. These data were publicly available from the CCD and state education 
agencies for 100 percent of the study schools (U.S. Department of Education, 2015). The 
data collected include the following: 

• School size. School size data included the total number of grade 9 students and the 
total student enrollment for each school from the CCD for the 2012/13 school 
year (U.S. Department of Education, 2015). These 2012/13 data were used in the 
assessment of baseline equivalence and as covariates in the regression analyses 
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for treatment contrast and school data culture. The study team also used public- 
ly available grade 9 enrollment records retrieved from the three state education 
agency websites for the 2011/12 school year for random assignment because these 
were the most recent data on school size available. 

• On-time graduation rates. On-time graduation rate data included four-year high 
school graduation rates from 2011/12 and 2012/13 for each study school from the 
websites of the three state education agencies. The study team used on-time gradu- 
ation rates from 2011/12 for random assignment because they were the most recent 
data available. The on-time graduation rates from 2012/13 were used to examine 
the equivalence of EWIMS and control schools at baseline, before training began 
for the EWIMS schools in April 2014. 

• Percentage of students who were eligible for the federal school lunch program. The total 
number of students eligible to participate in the federal school lunch program was 
collected for the 2012/13 school year from the CCD (U.S. Department of Educa- 
tion, 2015). The percentage for each school was calculated by dividing the number 
of eligible students by the school’s total student enrollment. 

• Percentage of students who identified as a member of a racial/ethnic minority. Data 
on race/ethnicity for the 2012/13 school year were collected from the CCD (U.S. 
Department of Education, 2015). The percentage of racial/ethnic minorities for each 
school was calculated by summing the total number of students who identified as 
American Indian/Alaska Native, Asian, Black, Hawaiian Native/Pacific Islander, 
Hispanic, or two or more races and dividing by the school’s total student enrollment. 

School leader survey. A web-based survey was administered to one school administrator 
at each EWIMS school and control school at the end of the 2014/15 school year. Surveys 
were also administered in spring 2014, but these data were used only to gather information 
on EWIMS schools’ interventions. The 2015 surveys were used to do the following: 

• Measure school data culture. 

• Collect information about the interventions used to support at-risk students. 14 

• Document the treatment contrast between EWIMS and control schools. 

School leaders who were the designated contacts for the study or who were knowledge- 
able about the interventions available for at-risk students at their school were asked to 
participate in the survey. Respondents included principals, assistant principals, guidance 
counselors, and school coordinators. The school leader survey response rate in spring 2015 
was 90 percent (66 schools). 15 

School data culture measures. School data culture refers to the ways schools use data to 
inform decisions about supports for students. Items from an earlier study about the use 
of benchmark assessment data to inform instructional decisions (Faria et al., 2012) were 
adapted for use in the current study. The school leader survey included 53 items, most with 
4-point response scales, as well as some frequency count questions to measure the frequen- 
cy of data use practices. The survey included four subscales: context for data use, concrete 
supports for data use, data-driven student support, and barriers to data use. 

• Context for data use. School goals for data use and the professional climate around 
data use. 

• Concrete supports for data use. The presence or absence of structured time to 
review data, training and professional development about data use, the availability 
of data coaching, and principal leadership. 
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• Data-driven student support. The frequency, duration, mode, and type of data 
review, as well as the ways teachers and schools respond to student data, includ- 
ing assigning general and targeted interventions, determining teacher professional 
development needs, and informing school improvement plans. 

• Barriers to data use. Staff and logistical barriers to data use, such as lacking time 
to review data or not having access to data that are timely and actionable. 

Rasch analyses for ordered response categories (Rasch, 1960 [1980]) were used to determine 
whether the survey items, as well as each subscale, reliably measured the latent constructs 
of overall school data culture. The Rasch approach differs from classical test theory (aver- 
aging the percentage of respondents endorsing each response option) because the Rasch 
model considers the relative frequency with which each item and response option is used 
(that is, item difficulty). 16 The resulting scale scores provided a quantitative view of the 
frequency and intensity of respondents’ answers across a set of items representing school 
data culture. The subscales and overall measure of school data culture had high internal 
consistency, with Cronbach’s alpha reliability coefficients for the scales ranging from 0.80 
to 0.97 (table C3). 


Table C3. School data culture scale and subscales during the 2014/15 school year 

Scale and 
subscales 

Number 
of items 

Example items 

Example response options 

Cronbach s 
alpha 

School data culture 

53 



0.97 

Context for data 

use 

9 

How much do you agree 
or disagree with the 
following statements 
about how your school 

uses data? This school 

has clear goals for using 
data to improve student 

outcomes. 

■ Strongly agree 

• Agree 

• Disagree 

■ Strongly disagree 

0.81 

Concrete supports 

for data use 

12 

In the last 12 months, how 
much did the professional 
development at your 
school emphasize the 
following? 

Using data to target 

interventions for low- 

performing 

students. 

• Major emphasis 

• Moderate emphasis 

• Minor emphasis 

• No professional 
development provided 
on this topic 

0.92 

Data-driven 
student support 

23 

Please indicate the extent 
to which your teachers 
do each of the following: 
meet together to look 
at trends in the data (or 
analyze data). 

• To a great extent 

• To a moderate extent 

• To a slight extent 

• Not at all 

0.92 

Barriers to data 

use 

9 

To what extent do the 

following factors hinder 
your ability to use student 

data to inform instruction 

and interventions? Lack 

of time to study and think 

about available data. 

• To a great extent 

• To a moderate extent 

• To a minor extent 

• Not at all 

0.80 

Note: The school data culture instrument can be obtained by contacting the first author of this report. 

Source: Authors' analysis based on school leader survey. 


C-6 




Extant documents on EWIMS implementation from the technical assistance liaisons. 

Extant data from the EWIMS technical assistance liaisons included attendance sheets 
from all trainings, completed school implementation summaries, and customized support 
plans. Attendance sheets were used to examine EWIMS schools’ participation at each pro- 
fessional development session. Each EWIMS school’s technical assistance liaison complex 
ed an implementation summary at the end of the 2014/15 school year for each school that 
did not cease implementation. The school summaries and customized technical assistance 
plans provided information on the successes and challenges schools experienced with each 
step of the EWIMS seven-step process and a summary of the technical assistance provided 
for each school. Extant documents from implementation were used to examine the level 
of implementation at each participating EWIMS school (using a rubric described in the 
“Implementation analyses” section later in this appendix). 

EWIMS training satisfaction surveys. Satisfaction surveys were collected after each 
training to document satisfaction with EWIMS implementation. The satisfaction surveys 
gauge participants’ satisfaction with the quality, relevance, and usefulness of the service 
provided. Satisfaction surveys for the in-person trainings (regional, early warning data tool 
training, and site visits one through three) were paper based and administered in person. 
Satisfaction surveys for the online trainings (early warning data tool refresher and Web- 
Shares one through five) were web based and administered electronically. 

The number of participants expected at each type of training varied. The expectation for 
on-site, in-person trainings (regional trainings and site visits) was that the full EWIMS 
team would participate. Satisfaction survey response rates for training activities with 
EWIMS teams are presented at the individual participant level (table C4). 

Response rates for satisfaction surveys from the in-person trainings ranged from 60 percent 
to 88 percent across all schools, with the regional training having the highest response rate 
(see table C4). The response rates were calculated from the number of schools where at 
least one participant responded to the survey. The expectation for tool trainings and Web- 
Shares was that at least one representative from each school would participate. Response 
rates for the training activities for which only one school representative was expected to 
participate ranged from 86 percent (WebShare 1) to 97 percent (early warning data tool 
training; table C5). 

EWIMS monthly log data. EWIMS teams at each school were asked to keep logs of the 
team meetings as part of typical implementation, and the expectation was that teams were 
meeting monthly. Meeting logs were collected from the EWIMS schools at the end of 
each month to document the frequency and content of meetings. EWIMS schools were 


Table C4. Participant-level satisfaction survey response rates for on-site visits with 
Early Warning Intervention and Monitoring System teams in the 2014/15 school year 


Training 

Timing 

Number of completed 
satisfaction surveys 

Total number of 
participants 

Response rate 
(percent) 

Regional training 

Spring 2014 

173 

196 

88.3 

Site Visit 1 

Summer 2014 

174 

224 

77.7 

Site Visit 2 

Winter 2014-15 

84 

139 

60.4 

Site Visit 3 

Spring 2015 

108 

163 

66.3 

Source: Study records. 
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Table C5. School-level satisfaction survey response rates in the 2014/15 school year 

Training 

Timing 

Number of 
completed 
satisfaction 
surveys 

Total 

number of 
participants 

Response 

rate 

(percent) 

Early warning data tool training 

Spring 2014 

35 

36 

97.2 

Early warning data tool refresher training 

Fall 2014 

32 

35 

91.4 

WebShare 1 

Fall 2014 

24 

28 

85.7 

WebShare 2 

Winter 2014-15 

20 

23 

87.0 

WebShare 3 

Winter 2015 

20 

22 

90.9 

WebShare 4 

Spring 2015 

19 

22 

86.4 

WebShare 5 

Spring 2015 

18 

19 

94.7 

Source: Study records. 


provided a log template they could use for their EWIMS team meetings, and the meeting 
logs recorded the meeting participants, agenda, meeting activities, and action items. Most 
schools opted to use the meeting log template, but some schools provided notes from the 
meeting instead of or in addition to the meeting log. 

Between June 2014 and June 2015, 35 EWIMS schools (95 percent) held at least one 
EWIMS team meeting, and 33 schools (89 percent) submitted at least one meeting log. In 
total, 217 EWIMS team meetings were held throughout the implementation period, and 
monthly logs were collected from 207 team meetings (95 percent). 

Early warning data tool reports. Reports generated from the early warning data tools 
captured tool use and were requested from EWIMS schools during the 2014/15 school year 
after each quarter, trimester, or semester, depending on each school’s grading period. The 
tool produced four summary reports that were exported as xml files: 

• Student Flag Report. This report displayed a list of all students identified for differ- 
ent indicators of risk: chronic absence, course failure, low GPA, or behavior (e.g., 
suspensions). For each indicator, a student was flagged by the tool if the student 
had data imported into the tool and met the research-based indicators of risk pro- 
grammed into the tool. 

• Data Import Report. This report recorded the date and time when files were import- 
ed by a school into the early warning data tool. Importing student demographic 
data and initial data on output indicators is a required step prior to using the tool. 

• Student Intervention Report. This report contained each student-specific interven- 
tion assigned to a student in the early warning data tool. The report included 
the intervention name, type (for example, academic, behavioral, chronic absence), 
and the start and end dates for each student assigned to the intervention. 

• Student Data Report. This report consisted of the actual student data uploaded 
into the tool, including student demographics (gender, race/ethnicity, identified 
disability, and English learner and economic status), incoming risk flag, atten- 
dance, course failure, GPA, behavioral, and cumulative credits data. This report 
documented the completeness of the data that schools uploaded into the tool 
(that is, the percentage of students for whom demographic and administrative data 
records were uploaded). 

The most recent early warning data tool reports for the 2014/15 school year were used 
for the implementation analyses. Thirty-one EWIMS schools (84 percent) shared all four 
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tool reports at least once. Data from the early warning data tool were coded according to 
the implementation rubric, detailed in the “Implementation analyses” section later in this 
appendix. 

EWIMS team interview. In spring 2015, EWIMS team members were interviewed to 
collect qualitative information about how each school implemented the EWIMS model. 
The interview protocol covered each step of the seven-step implementation process and 
asked questions about the progress, successes, and challenges encountered at each step. 
The EWIMS team chairperson, the data designee, and up to one other team member were 
invited to participate in the interview. Twenty-eight of the EWIMS teams participated in 
the interview (76 percent). 

Exit interviews were conducted with seven of the eight schools that stopped, or never 
began, implementing EWIMS during the 2014/15 school year; the one school that dropped 
out of the study after random assignment but before implementation was not interviewed. 
These schools received a modified version of the team interview that focused on the 
reasons why they chose not to implement EWIMS. Each interview was transcribed and 
coded according to the implementation rubric, described in the “Implementation analyses” 
section later in this appendix. 

Treatment contrast. The study was designed to minimize the use of an early warning 
system similar to EWIMS in control schools in multiple ways. These included screening 
out schools from the study sample that used an early warning system or used data in ways 
that mirrored the EWIMS seven-step process, as well as giving all control schools access 
to all features of EWIMS the following school year at no cost (a delayed treatment design). 
Also, attendance records from EWIMS trainings demonstrate that no control schools 
accessed the professional development offered in the study. Nevertheless, it is still impor- 
tant to describe whether and how EWIMS and control schools differed in their practices 
related to identifying and supporting at-risk students during the year of the evaluation. 

This study did not collect student-level information about the treatment contrast (for 
example, detailed data in control schools about practices related to reviewing student data, 
assigning interventions, and monitoring student progress). The process of tracking this 
information would resemble the implementation of EWIMS, so that collecting these data 
might have diminished the contrast between EWIMS and control schools. 

Instead, the treatment contrast was measured with the 2015 school leader survey using 
items that captured the extent to which schools adhered to the practices of their randomly 
assigned groups by using or not using an early warning system. Beyond self-reported use of 
an early warning system, the study also examined contrasts between EWIMS and control 
schools in some early warning system-related practices, including whether schools report- 
ed having a dedicated team to identify and support at-risk students, school self-reported 
frequency of attendance and course failure data review, and the number and type of inter- 
ventions offered to students. Leaders from 67 of the 73 schools (90 percent) responded to 
these items. 

To measure the extent to which schools adhered to their randomly assigned groups, the 
survey asked school leaders to indicate whether they used an early warning system. The 
survey wording was as follows: 
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Is your high school using an early warning system to identify students who may be at risk 
of not graduating from high school on time? Response options were as follows: 

• Yes, we are using an early warning system at my high school. 

• No, my high school is not currently using an early warning system. 

All school leaders participated in presentations during the recruitment process that pro- 
vided an overview of early warning systems and EWIMS before signing memoranda of 
understanding to adopt EWIMS in either the treatment or delayed-treatment condition. 
Specifically, school leaders received materials indicating that early warning systems rely 
on readily available data housed at the school to help predict which students are at risk of 
not graduating on time; target resources to support off-track students while they are still 
in school, before they drop out; and examine patterns to identify potential school climate 
issues. The materials further indicated that early warning systems use key indicators of 
engagement (attendance), course performance (course grades, credits earned, GPA), and 
behavior (suspensions) to flag at-risk students. 

The survey also asked school leaders to report whether or not they had a dedicated team to 
identify and support at-risk student. The survey wording was as follows: 

Does your school have a team or group of individuals that reviews student data to support stu- 
dents who are identified as at risk of not graduating from high school (for example building - or 
teacher-level teams, student success teams, data review teams)? The response options were as 
follows: 

• Yes, we have a dedicated school-based team. 

• No, we do not have a dedicated school-based team. 

• Other. 

In the memoranda of understanding, under roles and responsibilities, all schools partici- 
pating in the study agreed as part of the EWIMS implementation “to develop an EWIMS 
team within their school” that would be responsible for “identifying students who are at 
risk and ensuring that their individual needs are met through school-based interventions.” 
Therefore school leaders who responded to the survey, unless these were new due to staff 
turnover, had shared definitions of an early warning system and the school-based team to 
identify and support at-risk students as conceptualized for this study. However, one lim- 
itation of the survey item assessing whether schools had a school-based team was that the 
item did not provide a response option for a nondedicated school-based team. 

Items used to measure frequency of review of attendance and course performance data for 
treatment contrast analyses are shown in table C6. While both items were measured on 
ordinal response scales, they were operationalized into a binary indicator of whether or not 
schools reviewed these data at least monthly. The monthly cut-off was most appropriate 
given the frequency with which data are made available and the expected frequency of 
EWIMS team meetings (monthly). 
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Table C6. Survey items regarding frequency of data review used in the 2014/15 
school leader survey 


I Survey item 

Response options 

Does your school review student attendance data to 
determine which students may be at risk (that is, missing 
more than a certain number of days per year)? 

• Yes 

• No 

Does your school review student course failure data 
(including course failures, credit deficiencies) to determine 
which students may be at risk? 

• Yes 

• No 

How often do you review these [attendance] data? 

• Daily 

• Weekly 

■ Monthly 

• Four times per year (once per quarter) 

• Three times per year 

• Two times per year 

• One time per year 

How often do you review these [course failure] data? 

• Daily 

• Weekly 

• Monthly 

• Four times per year (once per quarter) 

• Three times per year 

• Two times per year 

• One time per year 

Source: Authors' compilation. 


The school leader survey also asked about the types of dropout prevention interventions 
and strategies that were available for students in EWIMS and control schools. These items 
were based on the constructs presented in Dropout Prevention: IES Practice Guide (Dynar- 
ski et al., 2008) and Approaches to Dropout Prevention: Heeding Early Warning Signs with 
Appropriate Interventions (Kennedy & Monrad, 2007). They survey items asked respom 
dents to provide the name and focus (chronic absence, course failure, behavioral, or a com- 
bination) of each intervention (table C7). 

The number of interventions offered in EWIMS and control schools was tallied using the 
data collected in the school leader survey and summarized in two ways. First, to generate a 
count of the number of interventions offered, the number of interventions used to address 


Table C7. Survey items used to document number and type of interventions used in 
the 2014/15 school leader survey 


1 Survey item 

Response options 

' ' ~~ ' 

Program name 

Fill in the blank 


Specific focus of the intervention 
(for example, math, Algebra 1 
remediation, reading, study skills, 
social emotional, peer mentoring, 
attendance, mental health) 

Fill in the blank 


Type of intervention 

• Targeted academic interventions 
- Targeted behavior interventions 

• Attendance or truancy interventions 

• Online content recovery programs 

• Mentoring programs 

• Internship or school-related 
work-preparation programs 

• College preparation 

Source: Authors' compilation. 
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chronic absence, academic performance, and behavioral issues was summed. Each EWIMS 
and control school was also coded as either offering interventions supporting all three 
categories of risk or as not offering them. 

Operationalizing student outcome data 

Main student outcomes. To create the main studentdevel outcome measures, the extant 
student data were coded as 1 or 0, reflecting the presence or absence of an indicator of risk 
of a student’s not graduating on time or not progressing in school. The thresholds used to 
create the binary indicators for chronic absence, course failure, low GPA, and suspensions 
were based on the thresholds that are used in the early warning data tool to identify stu- 
dents as at risk of not graduating on time (Heppen & Therriault, 2008; Therriault et al, 
2010). The thresholds for progress were established for the study. Each outcome variable 
was coded into the binary indicator used in the analyses testing the primary impact ques- 
tions (table C8). 

Outcomes not measured or measured with limitations. States, districts, or schools 
considering investing in an early warning system such as EWIMS may be interested in 
knowing the impact of the model on a number of additional outcomes that were beyond 
the scope of this study or were available on only a limited basis. These outcomes include 
persistence in school, dropout rates, student performance on state standardized assess- 
ments (for example, on exit exams or state assessments), grade promotion or multiyear 
credit accrual, and on-time graduation. 

Student persistence in school is part of the EWIMS theory of action and was intended 
to be a main outcome. However, due to limitations with the within-school year enroll- 
ment and exit code data available to operationalize persistence, it was more appropriate 
to present descriptive statistics on preliminary persistence and not treat the study’s per- 
sistence measure as a primary outcome. 

Preliminary persistence was operationalized for the study as enrollment status as of spring 
2015, using enrollment status and exit code data collected from states and schools at the 
end of the 2014/15 school year. These data included the date a student entered the school 
or school district (entry date), the date a student was no longer enrolled at the school (exit 
date), and the exit code recording the reason a student withdrew from the school. Enroll- 
ment data from both the 2013/14 and 2014/15 school years were combined with 2014/15 
outcome data (courses, grades, attendance) to determine whether a student was enrolled 
in school at the end of the 2014/15 year. Using the data available at the end of the 2014/15 
school year, preliminary persistence was operationalized in the following way: 

• Students with exit codes indicating that the students were enrolled in a school 
were coded as 1. 

• Students with exit codes indicating that the students left and did not reenroll in 
any school were coded as 0. 

• Students with exit codes indicating that the students transferred out of the 
country or to other schools, either in- or out-of-state, with no transcript requests 
from the new schools, were coded as having an unclear enrollment status (as were 
students who had an exit code of homeschooled 17 or deceased or who had an exit 
date prior to the end of the school year but who were missing an exit code). 
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Table C8. Coding of outcome variables for the 2014/15 school year data 

Outcome 

Coded values 

Coding description 

Short-term binary risk indicators 

Chronic absence 

1 = missed 10 percent or 

more of enrolled school 

days, 0 = did not miss more 
than 10 percent of enrolled 
school days 

The total number of days/hours/periods a student was absent during the 

2014/15 school year was divided by the total number of days/hours/periods 
a student was enrolled during the school year to calculate the percentage of 
instructional time a student missed. Percentages greater than or equal to 

10 percent were coded as 1, and percentages less than 10 percent were coded 
as 0. Students with an exit code that clearly indicated dropout and missing data 

for the chronic absence outcome were coded as 1. 

Course failure 

(failing any course) 

1 = failed one or more 

courses, 0 = did not fail any 

courses 

Course grades for all attempted semester-long, trimester-long, or yearlong 
courses during the 2014/15 school year were coded, and students who received 
one or more F’s or E’s were coded as 1. Students who did not receive any F's or 
E's were coded as 0. Students with an exit code that clearly indicated dropout 
and missing data for the course failure outcome were coded as 1. 

Low grade point 
average (GPA) 

1 = GPA of 2.0 or lower, 

0 = GPA above 2.0 

Cumulative GPAs on a 4.0 scale were recoded into a binary variable where 

GPAs lower than or equal to 2.0 were coded as 1 and GPAs higher than 2.0 
were coded as 0. Four schools (2 EWIMS schools and 2 control schools) used 
a 12-point GPA scale. GPAs on a 12-point scale were coded as 1 if the GPA was 
lower than or equal to 6.0, and GPAs above 6.0 were coded as 0. There were no 
differences in the number of EWIMS and control schools that provided cumulative 
versus noncumulative GPA data (two EWIMS schools and no control schools 
provided GPAs only for the 2014/15 school year — that is, noncumulative GPAs). 
Cumulative and noncumulative GPA values were combined for analyses. Students 
with an exit code that clearly indicated dropout and missing data for the low GPA 

outcome were coded as 1. 

Suspension 

1 = suspended one or more 
times, 0 = not suspended 

Total suspensions during the 2014/15 school year were coded as 1 if the number 
of suspensions was greater or equal to 1 or if the number of days suspended 
was greater or equal to 1. Total suspensions included in-school and out-of- 
school suspensions. Students with zero suspensions or missing information on 
suspensions were coded as 0 if behavioral data were provided for the school. 

That is, students missing behavioral data were assumed to have no behavioral 
incidents, after confirming with schools. Students with an exit code that clearly 
indicated dropout and missing data for the suspension outcome were coded as 1. 

Student progress in school 

Progress 

1 = insufficient credits 

earned, 0 = sufficient 
credits earned [number of 
credits needed to graduate 
divided by two for second- 
year students and by four 
for first-year students] 

In State 1, first-year students with 10 or more credits and second-year students 
with 20 or more credits were coded as 0. In State 2, first-year students with 4.5 
or more credits and second-year students with 9 or more credits were coded as 0. 
In State 3, first-year students with five or more credits and second-year students 
with 10 or more credits were coded as 0. Students in each state and grade 
below these thresholds were coded as 1. Students with an exit code that clearly 
indicated dropout and missing data for the progress outcome were coded as 1. 

EWIMS is the Early Warning Intervention and Monitoring System. 

Source: Authors' compilation. 



The data used to operationalize preliminary persistence are potentially incomplete or 
not fully accurate because schools often report students as still enrolled at the end of the 
school year even if they have not attended recently and wait to assign a more formal exit 
code (for example, transfer, dropout) in the fall of the next school year if the student fails 
to matriculate. Because this study did not follow students through fall 2015, persistence 
rates in spring 2015 may appear higher than they were and may not yet reflect potential 
differences by condition. For this reason, findings involving preliminary persistence should 
be interpreted with caution, and a longer study is needed to more accurately estimate the 
impact of EWIMS on school persistence. Analyses on preliminary persistence can be 
found in appendix D (see figure D2). 
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A longer study is also needed to estimate the impact of EWIMS on dropout rates from year 
to year, as well as on grade promotion and multiyear credit accrual. Those outcomes that 
require longitudinal data collection across school years were out of scope for the current 
study but are acknowledged as critically important outcomes that an intervention like 
EWIMS intends to positively affect. 

Student performance on state standardized tests also was originally intended to be includ- 
ed as a main study outcome. However, because the states involved in the study planned to 
transition to different state assessments (aligned with the Common Core state standards) 
in spring 2015, studentdevel state test scores were not available for use in the study. In 
addition, only one of the states used exit exams. 

Finally, as the purpose of EWIMS is to identify and support at-risk students to help them 
get on track for on-time graduation, graduation within four years of high school entry is a 
critical outcome, but it was out of scope for this particular study. As noted in the Implica- 
tions section and throughout the main report, additional research is needed to document 
the potential impact of EWIMS on on-time graduation rates. 

Impact analyses 

Impact analyses of student outcomes. Because the primary outcomes were binary and 
students were nested in schools, all primary impact models and sensitivity analyses used 
multilevel logistic regression models. Models assumed a constant treatment effect and 
included dummy variables to capture the matched pair random assignment. Baseline 
student demographic and prior achievement characteristics were also included to improve 
the precision of the treatment estimate. A multilevel logistic regression was used for the 
binary outcomes with the following formula: 

y] ; . = y 00 + y 01 (condition.) + y 02 X(pair ID dummy variables^) + 
y 10 X(student demographic covariates^) + y 20 (grade 8 reading test score ; .) + 
y 30 (grade 8 math test score ; .) + y 40 X(missing data flags i( ) + u Qj 

where vy.. = log (<p ; . / 1 - <p ; .) is the log of the odds of the binary outcome of interest. For 
example, v] y is the log odds of failing a course for student i in school j, and tp ; . is the proba- 
bility of failing a course. If the probability of failing a course <p ; . is 0.5, the odds of failing a 
course v) ; . = log (<p ; . / 1 - <p ; .) = 0.5 /0.5 = 1.0, and the log-odds (or logit) is log (a) = 0. If the 
probability of failing a course is less than 0.5, the odds are lower than 1.0 and the logit is 
negative. 

Condition, is an indicator variable that takes a value of 1 for EWIMS schools and 0 for control 

i 

schools; y 00 represents the log odds of the binary outcome of interest (for example, failing a 
course) when all covariates are 0; y 01 captures the impact of EWIMS (or the difference in 
log odds of the binary outcome of interest [for example, failing a course] between students in 
EWIMS schools and control schools, controlling for all student and school covariates); y 02 
represents the vector of coefficients for the matched pair dummy indicators; y 10 represents 
the vector of coefficients for the student demographic characteristics (including age, gender, 
racial/ethnic minority status, special education status, English learner status, and grade); y 20 
and y 30 represent students’ prior achievement on state math and reading tests; y 40 represents 
the vector of coefficients for the missing data flags; and u Q . is the error term for school j. 
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Standardized differences for binary outcomes are measured using the Cox index effect size. 
Cox index effect sizes are standardized differences in the probability of the occurrence of 
an event. They yield effect size values similar to the values of Hedges’ g that one would 
obtain if group means, standard deviations, and sample sizes were available, assuming the 
dichotomous outcome measure is based on an underlying normal distribution. 

Sensitivity analyses. In addition to estimating the primary impact models just described, 
the study conducted sensitivity analyses using different combinations of the studentdevel 
covariates, different strategies to adjust for missing data, and different specifications of the 
outcome variables. Sensitivity analyses included the following: 

• A set of models that excluded all student covariates but retained the dummy vari- 
ables for school matched pairs and the treatment indicator. 

• A set of models that excluded all student covariates, with the exception of the 
standardized grade 8 state reading and math test scores and the missing data flags 
for these two prior achievement measures but retained the dummy variables for 
school matched pairs and the treatment indicator. 

• A set of models that contained the same covariates as the primary impact model 
and used listwise deletion instead of dummy variable covariate adjustment for 
missing data. 

• A set of models that contained the same covariates as the primary impact model 
and used inverse probability weighting to adjust for missingness. 

• A set of models that used a common analytic sample for all outcomes (that is, 
schools and students with nonmissing outcome data for all outcome measures). 

• A set of models that used the continuous variables that were used to create the 
binary risk indicators (for example, GPA in place of low GPA). 

Two additional sensitivity analyses were conducted for specific outcomes. For the course 
failure primary impact model, one additional sensitivity analysis used a binary indicator 
of core course failure. For the low GPA primary impact model, one additional sensitivity 
analysis removed the four schools (and their matched pairs) that used a 12.0 scale for GPA. 

For the models that used continuous data rather than the binary risk indicators, linear 
multilevel regression models were specified. A continuous specification of chronic absence 
(percentage of instructional time absent), course failure (percentage of courses failed), GPA 
(on a 4.0 or 12.0 scale, depending on the school), and credits earned were regressed on the 
same set of covariates included in the primary impact models just discussed. Because four 
schools calculated GPA on a 12.0 scale rather than a 4.0 scale, the results for GPA were 
estimated by including a dummy indicator for whether the school had a 12.0 scale or not. 
These models were estimated using the following equation: 

Y,j = 7oo + 7oi( con dition.) + y 02 (pair ID dummy variables^ + 
y 10 (student demographic covariates ; .) + y 20 (grade 8 reading test score ; .) + 
y 30 (grade 8 math test score.) + y 40 (missing data flags ; .) + u Q . + r.. 
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Analyses by grade. Subgroup analyses examined whether EWIMS had a different effect 
for students in grades 9 and 10. These analyses used models that interacted condition with 
grade level, as displayed in the following model: 

% = Too + 7oi( c °ndition. * gradelO y ) + y 02 (condition. * (l-gradelO^)) + 
y 03 S(Pair ID dummy variables^ + y 1Q student demographic covariates^) + 
y 20 (grade 8 reading test score fj ) + y 30 (grade 8 math test score i( ) + 
y 40 (missing data flags y .) + y n (gradel0..) +u 0j . 

where y 01 represents the coefficient for the treatment effect for second-year students and 
y 02 represents the coefficient for the treatment effect for first-year students. 

Exploratory school data culture analyses. Linear regression models were estimated to 
analyze the impact of EWIMS on school data culture, both the overall scale score and 
the school data culture subscales (context for data use, concrete supports for data use, 
data-driven student support, and barriers to data use). These analyses were considered 
exploratory because they had limited statistical power to detect an impact of EWIMS at 
the school level, requiring an effect size of 0.70 or greater. These models included fixed 
effects for matched pairs, three covariates, and the treatment indicator, as follows: 

School data culture = (3 0 + (^(condition) + ^(school size) + (J 3 (on-time graduation rate) + 
(3 4 (baseline data-driven dropout prevention) + (3 5 (pair ID Dummy Variables) + £ 

School data culture is the school-level Rasch score on the data culture survey scale. The 
parameter of interest, (Jj, estimates the impact of EWIMS on school data culture. Similar 
models were run to examine the impact of EWIMS on the four data culture subscales that 
composed the overall data culture measure. The effect size for each school-level analysis 
was calculated as Hedges’ g, which adjusts for small sample sizes. 18 

Implementation analyses 

Participation in professional development offered to EWIMS schools. Attendance sheets 
were used to calculate the percentage of the 37 EWIMS schools that participated in each 
session (see appendix A). 

Level of implementation of the EWIMS seven-step process overall and for each of the 
seven steps (programmatic implementation). Analysis of the extent and level of imple- 
mentation of the seven steps of the EWIMS process was based on a rubric that drew on 
multiple data sources. The rubric measured schools’ levels of implementation on each of 
the seven steps of the EWIMS process and across the seven steps for an overall implemen- 
tation rating (table C9). To construct the rubric, the EWIMS technical assistance liaisons 
identified key features of implementing each step; set thresholds for low, moderate, and 
high levels of implementation on each indicator; and established requirements for achiev- 
ing low, moderate, or high implementation overall and for each step. The indicators for 
each step were based on the Early Warning System Implementation Guide developed by 
the National High School Center (Therriault et al., 2010). To achieve moderate or high 
implementation scores across steps, the rubric required consistently high or moderate 
implementation ratings on each of the EWIMS steps, except step 7 (evaluating and refin- 
ing the EWIMS process). 
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Table C9. Rubric used to measure implementation during the 2014/15 school year 

Step 1: Establish roles and responsibilities 

Step 1 involves organizing the personnel required to employ the process in the school. Schools may construct a new school based team to run EWIMS, or the process may 
be incorporated into an existing team. The EWIMS team is responsible for implementing the EWIMS process in the school, and EWIMS team members are the recipients 
of EWIMS training. Ideally, EWIMS team members reflect a range of responsibilities, skill, and knowledge. The team should have the authority to make decisions for the 
building, and have access to information about individual students. The team is expected to meet regularly (at least monthly) throughout the school year. 

Indicator 

“Low” rating definition 

"Moderate” rating definition 

“High” rating definition 

1.1 Champion 

No EWIMS champion on team or 

The team has a champion, but he or she is not empowered 

The team has a champion. This champion has the 


in the school building. 

to make changes 

authority to implement new interventions and make 



OR 

The team has an empowered school leader on the team, 
but this leader is not invested in the EWIMS process. 

changes within the school building. 

1.2 Team membership 

The team lacks members with 

The team includes staff members who have knowledge 

Team includes staff members with knowledge of students 


knowledge of interventions and 

of students or knowledge of interventions available to 

and the interventions available to those students. 


student needs. 

students. 


1.3 Meeting frequency 

The team does not formally 

The EWIMS team holds formal meetings at least every 

The EWIMS team holds formal meetings on a monthly 


meet or holds three or fewer 

other month (four to seven meetings over the year), 

basis OR the team holds meetings in every month but 


formal EWIMS meetings during 

and these meetings align to major points in the school 

one. 


the course of the school year. 

calendar. 


Step 1 overall 

School receives a “low” rating 

School receives at least a “moderate" rating on each 

School receives “high" ratings on Champion and Team 


on at least one indicator. 

indicator, but does not do well enough to be rated “high” 

Membership indicators and no lower than "moderate" on 



overall on this step. 

Meeting Frequency indicator. 

Step 2: Use the early warning data tool 

Step 2 involves the deployment of technology to identify students who are at risk of not graduating high school on time. Staff are expected to import both student- and 
intervention level data into the tool at prescribed times (before the school year, at the 20- or 30 day mark, at the end of each grading period, and at the end of the year). Staff 
also should upload and keep updated an inventory of school based interventions, and should use the tool to document the assignment of students to interventions (including 
start and end dates). 

Indicator 

“Low” rating definition 

“Moderate” rating definition 

“High” rating definition 

2.1 Timing of imports 

Does not import student data 

Imports student data at least once during the school year, 

Imports student data into the tool throughout the school 


into the early warning data tool. 

apart from the training provided by the EWIMS technical 

year (once in the winter for schools on a semester 



assistance liaisons. 

schedule, once in the winter and spring for schools on 
trimester schedules). 

2.2 Correct and 

Does not import student data 

Imports student data into the tool, but it is not for the full 

Imports correct student data into the early warning data 

complete data 

into the early warning data tool. 

set of data elements — demographic, attendance, behavior, 
and academic data. 

tool. 

2.3 Tracking 

Zero or one interventions have 

Students are assigned to at least one intervention in the 

Schools use the early warning data tool to assign 

interventions 

been uploaded into the early 

tool, apart from assignments constructed as a part of the 

students to interventions in a systematic manner. 


warning data tool. 

early warning data tool training. 

Schools may assign interventions at several points 
during the school year, or the same intervention is 
updated with different entry and exit dates for students. 

Step 2 overall 

School receives a “low” rating 

School receives at least a “moderate" rating on each 

School receives a “high” rating on each indicator. 


on at least one indicator. 

indicator, but does not do well enough to be rated “high” 
overall on this step. 



(continued) 
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Table C9. Rubric used to measure implementation during the 2014/15 school year (continued) 

Step 3: Review the early warning data 

In step 3, EWIMS teams review data from the early warning data tool to identify students who are not on track for graduation. In addition, the EWIMS team uses the 
early warning data tool to understand grade- and building level patterns in student risk. Schools should go through step 3 whenever new data are added to the tool. 

Indicator 

"Low” rating definition 

"Moderate" rating definition 

“High” rating definition 

3.1 Identification of at- 

risk students 

Team does not use a systematic 
approach to identify at-risk 

students. 

Team employs a systematic approach to identifying 

students but uses nonresearch-based thresholds to 

identify students at risk of not graduating on time 
(excluding locally validated indicators). 

Team uses research-based or locally validated thresholds 
to identify students at risk of not graduating on time. 

3.2 Frequency of 

student data review 

Team reviews student data in 

fewer than three EWIMS team 
meetings per school year. 

Team reviews student data in three to five EWIMS team 

meetings per school year. 

EWIMS team reviews student data in more than five 

EWIMS team meetings per school year. 

3.3 Reviewing patterns 
of risk 

Does not use early warning 
system data to explore school- 
and grade-level patterns of risk. 

Uses early warning system data to explore trends and 
patterns in student risk but does not generate additional 
questions about student needs. 

Uses early warning system data or reports to explore 
school- and grade-level trends and patterns of risk and 
uses this data as starting points to generate questions 

about student needs. 

Step 3 overall 

School receives a "low” rating 

on at least one indicator. 

School receives at least a “moderate" rating on each 
indicator, but does not do well enough to be rated “high” 
overall on this step. 

School receives a “high" rating on each indicator. 

Step 4: Interpret the early warning data 

In step 4, the EWIMS team moves from who is flagged as off track to why those students were flagged. Teams engage in the analysis of student data present in the tool and 
consider information and data from outside of the tool (that is, other teachers, conversations with parents, assessment data) to understand the root causes of students’ risk 
status. The early warning data tool contains reports designed to facilitate this process. 

Indicator 

“Low” rating definition 

"Moderate” rating definition 

“High” rating definition 

4.1 Introduce 

supplemental data 

Does not incorporate other 
information (other data, 
personal knowledge, other 
staff) to diagnose student 

needs. 

Uses data and knowledge “in the room” to determine 

student needs. 

Incorporates several sources of additional information to 
diagnose student needs. 

4.2 Understanding 
underlying causes 

Does not examine underlying 

causes of student risk. 

Sometimes examines underlying causes of risk, but this 
process is not used consistently or is unstructured. 

Team uses a systematic process to explore underlying 
causes of risk for flagged students in order to assign 
them to the appropriate intervention(s). 

Step 4 overall 

School receives a “low” rating 

on at least one indicator. 

School receives at least a “moderate” rating on each 
indicator, but does not do well enough to be rated “high” 
overall on this step. 

School receives a “high” rating on each indicator. 


(continued) 
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Table C9. Rubric used to measure implementation during the 2014/15 school year (continued) 

Step 5: Assign and provide interventions 

Once the EWIMS team discusses the root causes of flagged students' risk indicators, they should match those students to the intervention or resource that will be best 
suited to support the students. The school not only should have many different kinds of interventions available to students, but school staff members (especially on the 

EWIMS team) need to know which interventions will address each student s (or groups of students ) needs. 

Indicator 

“Low” rating definition 

"Moderate” rating definition 

“High” rating definition 

5.1 Inventory of 

School does not have an 

School has an inventory of interventions, but these 

School has interventions available to students that 

interventions 

inventory of interventions. 

interventions do not address all three factors of risk 

address all three risk factors. 



(attendance, behavior, academics). 


5.2 Assignment 

Team does not match students 

Team matches students to interventions but only at 

Team matches students to interventions on a rolling, as- 

of interventions — 

to interventions. 

infrequent time points, such as the beginning of the year 

needed basis. 3 

frequency 


or the end of grading periods. 


5.3 Assignment 

Team matches less than 

Team matches 25 percent to 74 percent of identified at- 

Team matches at least 75 percent of identified at-risk 

of interventions — 

25 percent of identified at-risk 

risk students to interventions. 

students to interventions. 

coverage 

students to interventions. 



Step 5 overall 

School receives a “low” rating 

School receives at least a “moderate" rating on each 

School receives a “high" rating on Inventory of 


on at least one indicator. 

indicator, but does not do well enough to be rated “high” 

interventions and Assignment of interventions — frequency, 



overall on this step. 

and at least a “moderate” rating on Assignment of 
interventions — coverage. 

Step 6: Monitor students 

Once the EWIMS team has assigned students to interventions, the team should track those students’ progress in those interventions. The team examines whether students 
appear to be responding positively to the intervention (and moving toward being on track) or if the intervention does not appear to be working. In addition, the team monitors 
student progress to consider the overall effectiveness of interventions. 

Indicator 

“Low” rating definition 

"Moderate” rating definition 

“High” rating definition 

6.1 Track student 

School does not monitor 

Team monitors student progress in interventions and 

Team monitors student progress on an ongoing basis 

progress in 

student progress in 

decides whether to continue placement or modify the 

(at least monthly) and incorporates data into the 

interventions 

interventions, and assignment 

assignment, but the decision does not incorporate data. 

decision whether to continue or modify the intervention 


remains static. 


assignment. 3 

6.2 Monitor 

School does not consider 

School modifies interventions without using data to 

School uses data to consider the effectiveness of 

effectiveness of 

modifying intervention inventory. 

consider effectiveness of those interventions 

interventions and may add, eliminate, or modify those 

interventions 


OR 

School uses data to consider intervention effectiveness 
but does not make changes to intervention inventory. 

interventions. 

Step 6 overall 

School receives a "low” rating 

School receives at least a "moderate" rating on each 

School receives a “high” rating on each indicator. 


on at least one indicator. 

indicator, but does not do well enough to be rated “high” 
overall on this step. 



(continued) 
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Table C9. Rubric used to measure implementation during the 2014/15 school year (continued) 


Step 7: Evaluate and refine the early warning data process 

Step 7 involves EWIMS team reflection about successes and challenges related to the previous six steps. The team should understand its current strengths and weaknesses 
and create documented plans for improvement. At the conclusion of the school year, the team should ensure a process is in place to transition to the next fall. 


Indicator 

“Low” rating definition 

“Moderate" rating definition 

“High” rating definition 

7.1 Reflection on team 

progress 

The team has not identified 

successes or challenges that 
emerged from their school’s 

EWIMS 

process. 

The team has identified challenges to the EWIMS process, 
but has not identified clear action steps to address these 
challenges. 

The team has identified challenges and clear action 
steps to address these challenges in the coming school 

year 

OR 

The team has already begun to implement changes to 
address challenges. 

Step 7 overall 

School receives a “low” rating. 

School receives a “moderate” rating. 

School receives a “high” rating. 

Overall implementation rating 


“Low” rating definition 

“Moderate” rating definition 

“High” rating definition 

Overall implementation 

School does not receive 

moderate or high overall rating. 

School receives at least a “moderate” rating on steps 1 
through 6 but can have a “low” rating for step 7. 

School receives a “high” rating on steps 1 through 6 but 
can have a “moderate” rating for step 7. 


EWIMS is the Early Warning Intervention and Monitoring System. 

Note: "Research-based” for the purposes of this study was defined as schools using the indicators and thresholds preloaded into the tool to identify students at risk for not grad- 
uating on time. However, in other contexts, “research-based” could include locally validated indicators or thresholds based on historical data for that specific school or for similar 
schools, districts, or states. In interviews with EWIMS schools and the EWIMS meeting logs, EWIMS schools were asked to explain how they identified students at risk for not 
graduating on time, and some schools said they primarily used their own data or thresholds to identify students. Many of the schools in this group examined similar kinds of data 
(attendance records, behavior reports, and course failure reports), but they relied on past experience to decide which students to identify as at-risk students and assign them to 
interventions. A third group of schools relied exclusively on non-data-based sources (such as teacher referral) to identify students as at risk. These schools would be rated lower 
on step 3.1 (identification of at-risk students) because they were not using the research-based indicators from the tool (see table D7 in appendix D for a summary of the ratings on 
indicator 3.1). 

a. Although much of the data in the early warning data tool are refreshed each grading period, students could be assigned to new interventions on an as-needed basis based on 
attendance data, interim grades, or conversations with the student, family, or other school staff. Also, new students may enter the school who need to be assessed and assigned to 
interventions. 

b. EWIMS teams may monitor student progress in interventions using interim data, such as attendance reports, progress reports, or grades on tests or coursework, or they may 
regularly check in with students or school staff who work with those students. 

Source: Authors' compilation. 






The rubric was developed by the EWIMS technical assistance liaisons for the purpose of 
this project, but with full implementation of the model in mind. EWIMS is designed to 
be a multiyear continuous improvement process; therefore, many schools will not achieve 
high implementation of the full seven-step model in one year. For the study, placing 
EWIMS schools on a continuum reflecting a conceptualization of “full” implementation of 
the model was believed to most appropriately contextualize the impact analyses. 

To develop ratings for each EWIMS school in the study, two evaluators independently rated 
each school on all indicators within and across the seven steps by analyzing all available 
data that were collected from EWIMS schools, extant documents on EWIMS implemen- 
tation from the EWIMS technical assistance liaisons, monthly logs of EWIMS meetings, 
early warning data tool reports that captured tool use, and interviews with EWIMS team 
members conducted in spring 2015. 19 

Coders first independently coded each school and then met on a weekly basis to share 
their ratings, reconcile any discrepancies, and come to a consensus on a final implementa- 
tion score for each indicator and corresponding step. 

Barriers to implementation. To document the challenges that EWIMS schools faced in 
their first year of adoption, the study used the customized support plans and summaries 
that the EWIMS technical assistance liaisons created for each school. One analyst coded 
the customized support plans. Two analysts coded the school summaries, with each coding 
half of the summaries independently. These codes were developed iteratively based on 
themes that emerged regarding challenges (for example, compatibility between schools’ 
student information system and early warning data tool, schools’ capacity to manage and 
use data, schools’ ability to upload complete and accurate data to the tool, schools’ ability 
to use the tool’s intervention function, and challenges stemming from the amount of time 
it took to use the tool). Information from exit interviews from seven of the eight schools 
that stopped or never began implementing EWIMS also were used to describe the barriers. 

Specific types of interventions used by EWIMS schools and student assignment to 
interventions. To catalog the types of interventions offered in schools using EWIMS, 
the study used information from three data sources: the school leader surveys from spring 
2014 and 2015, 20 the early warning data tool reports, and the EWIMS team interviews. 
Interventions extracted from each data source were combined into one file, and duplicate 
interventions represented across multiple data sources for a school were removed. Each 
intervention was then coded by area of risk (chronic absence, behavior, course failure, and 
multidomain) and then by type of intervention (for example, academic support, mentor- 
ing, truancy, tutoring, and parent conference; see table D9 in appendix D). 

To examine the match between student indicators of risk and the interventions assigned 
to students, the study used data from the early warning data tools. Specifically, the number 
and percentage of students at risk because of chronic absence, course failure, or behavioral 
problems — or because of multiple risk indicators that were assigned to any intervention 
categorized as addressing those particular issues — were calculated. 

Treatment contrast. These analyses used items from the 2014/15 end-of-year school leader 
survey that asked about use of an early warning system, the presence of a dedicated team 
to identify and support at-risk students, the frequency of data review, and the number and 
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type of interventions. Responses to the survey were analyzed descriptively (the number 
and percentage of schools selecting each response option). To test the difference between 
EWIMS and control schools on measures of aspects of EWIMS implementation, a mix of 
logistic and linear regression models was conducted. The linear and logistic regressions for 
all treatment contrast analyses included three covariates — school size, on-time graduation 
rate, and baseline data-driven dropout prevention efforts. Linear regression also included 
dummy variables for matched pairs, but the logistic regressions do not include these vari- 
ables due to perfect prediction between the pair ID and binary outcomes. 

To measure whether schools adhered to their randomly assigned groups, the survey asked 
school leaders whether they used an early warning system during the 2014/15 school year; 
logistic regressions were run using binary indicators of whether or not a school reported 
having an early warning system. Logistic regressions were also run with a binary indicator 
of whether or not a school reported having a dedicated team to identify and support at-risk 
students. For the frequency of data review, logistic regressions were analyzed that used a 
binary indicator of whether or not schools reviewed attendance or course failure data at 
least monthly as the outcome. For the number of interventions, linear regressions were run 
that used the sum of the number of interventions reported on the school leader survey as 
the outcome. For type of interventions, logistic regressions were run using a binary indi- 
cator of whether a school offered each type of intervention (attendance intervention, aca- 
demic intervention, or behavioral intervention) as the outcome. 
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Appendix D. Detailed findings and supplementary analyses 


This appendix contains detailed findings related to the analyses presented in the main 
body of the report as well as supplementary analyses not included in the main body of the 
report. The analyses presented in this appendix include the following: 

Impact findings 

• Detailed findings from the main impact models and sensitivity analyses used to 
test whether the main impacts are robust to different model specifications for 
chronic absence, course failure, low grade point averages (GPA), suspensions, and 
progress in school. 

• Sensitivity analyses used to examine whether the impact of the Early Warning 
Intervention and Monitoring System (EWIMS) on continuous outcomes is consis- 
tent with the main impact analyses of binary outcomes. 

• Detailed findings for the exploratory analyses used to test whether the impact of 
EWIMS differed by grade (first-year students versus second-year students). 

• Supplementary analysis of the preliminary persistence measure. 

• Detailed findings for the exploratory analyses of school data culture. 

Implementation findings 

• Detailed findings on satisfaction with EWIMS training. 

• Detailed findings for the level of implementation. 

• Supplementary findings for the barriers experienced by schools that stopped or 
never began implementing EWIMS. 

• Detailed findings for the percentage of students flagged in the early warning tools. 

• Detailed findings for the specific types of interventions offered in EWIMS schools. 

• Detailed findings for the treatment contrast analyses. 

Detailed findings from the main impact models and sensitivity analyses used to test whether the 
main impacts are robust to different model specifications for chronic absence, course failure, low 
GPA, suspension, and progress in school 

Sensitivity analyses examined whether the impact estimates reported in the main body of 
the report were robust across different specifications of the impact models (tables D1 and 
D2). The sensitivity analyses suggest that the estimated impact of EWIMS on students is 
stable in magnitude, direction, and statistical significance, regardless of model specification 
for all student outcomes. 

Sensitivity analyses examining the impact of the Early Warning Intervention and Monitoring System 
on continuous rather than binary outcomes 

In addition to the sensitivity analyses that used alternate specifications of the analysis 
models, models were also analyzed that used the continuous specification of the binary 
outcomes. The continuous outcomes were the percentage of instructional time missed, the 
percentage of courses failed, GPA, and credits earned. In some cases, the student sample 
size for these models was larger than that used for the primary impact model because the 
models with continuous outcomes did not drop cases due to perfect prediction (see note in 
table Dl). 
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Table Dl. Results from main analyses and sensitivity models for chronic absence, course failure, low 
GPA, suspension, and progress in school in 2014/15 


Predicted probability 

Predicted probability 

Standard 


Cox index 

Student 

School 

Model 

for treatment 

for control 

Odds ratio 

error 

p-value 

effect size 

sample size sample size 

Outcome: Chronic absence (missed 10 percent or more of instructional time) 

1 

0.10 

0.14 

0.66 

0.07 

<.001 

-0.26 

35,876 

65 

2 

0.11 

0.15 

0.70 

0.08 

.001 

-0.21 

35,888 

65 

3 

0.10 

0.14 

0.66 

0.07 

<.001 

-0.25 

35,888 

65 

4 

0.09 

0.12 

0.77 

0.08 

.013 

-0.16 

25,732 

62 

5 

0.08 

0.11 

0.65 

0.08 

.001 

-0.26 

35,876 

65 

6 

0.09 

0.14 

0.64 

0.07 

<.001 

-0.28 

26,976 

52 

Outcome: Course failure (one or more course failures) 

1 

0.21 

0.26 

0.76 

0.06 

<.001 

-0.17 

35,133 

65 

2 

0.26 

0.30 

0.81 

0.06 

.002 

-0.13 

35,133 

65 

3 

0.22 

0.27 

0.76 

0.06 

<.001 

-0.17 

35,133 

65 

4 

0.20 

0.24 

0.81 

0.06 

.008 

-0.13 

25,294 

62 

5 

0.14 

0.18 

0.75 

0.07 

.002 

-0.17 

35,133 

65 

6 

0.21 

0.26 

0.73 

0.07 

.001 

-0.19 

26,976 

52 

7 

0.20 

0.24 

0.77 

0.06 

.001 

-0.16 

35,133 

65 

Outcome: Low grade point average (2.0 or lower) 

1 

0.17 

0.19 

0.87 

0.08 

.122 

-0.09 

30,080 

57 

2 

0.24 

0.26 

0.91 

0.08 

.278 

-0.06 

30,086 

57 

3 

0.18 

0.20 

0.86 

0.08 

.087 

-0.09 

30,086 

57 

4 

0.15 

0.16 

0.97 

0.09 

.731 

-0.02 

21,506 

54 

5 

0.15 

0.17 

0.86 

0.08 

.090 

-0.09 

30,080 

57 

6 

0.16 

0.18 

0.89 

0.09 

.249 

-0.07 

26,976 

52 

7 

0.17 

0.19 

0.85 

0.08 

.098 

-0.10 

26,829 

51 

Outcome: Suspension (suspended one or more times in the 2014/15 school year) 

1 

0.09 

0.09 

0.91 

0.13 

.497 

-0.06 

35,501 

63 

2 

0.11 

0.12 

0.93 

0.13 

.567 

-0.05 

35,558 

63 

3 

0.09 

0.10 

0.91 

0.13 

.497 

-0.06 

35,558 

63 

4 

0.08 

0.08 

1.10 

0.15 

.474 

0.06 

25,960 

61 

5 

0.08 

0.09 

0.89 

0.13 

.423 

-0.07 

35,501 

63 

6 

0.09 

0.10 

0.87 

0.15 

.420 

-0.08 

26,976 

52 

Outcome: Progress in school (credits earned) 

1 

0.14 

0.14 

0.98 

0.23 

.913 

-0.02 

35,044 

63 

2 

0.17 

0.17 

1.01 

0.19 

.967 

0.01 

35,050 

63 

3 

0.14 

0.15 

0.97 

0.22 

.878 

-0.02 

35,050 

63 

4 

0.13 

0.13 

1.00 

0.21 

.998 

<0.00 

24,949 

59 

5 

na 

na 

na 

na 

na 

na 

na 

na 

6 

0.09 

0.09 

1.07 

0.20 

.704 

0.04 

26,976 

52 


na is not applicable because the model with inverse probability weights did not converge for progress in school. 

Note: Model 1 is the primary impact model: a multilevel logistic regression with students at level 1 and schools at level 2, including binary 
indicators for student demographic covariates and grade, the grade 8 standardized tests in reading and math, missing data flags for each 
covariate, a set of dummy variables that captures school matched pairs, and the treatment indicator, which identifies schools as EWIMS 
schools or control schools. Model 2 has the same functional form as Model 1 but excludes all student covariates. Model 3 has the same 
functional form as Model 1 but excludes all student covariates except the pretests — the grade 8 standardized tests in reading and math 
and the missing data flags for these two pretest measures — and retains the dummy variables for school matched pairs and the treatment 
indicator. Model 4 has the same functional form as Model 1 and includes all of the covariates included in the primary impact model but 
uses listwise deletion for missing data on the covariates. Model 5 has the same functional form as Model 1 and includes ail of the co- 
variates included in the primary impact model but uses inverse probability weighting to correct for missingness on the outcome. Model 6 
has the same functional form as Model 1 but only includes students with nonmissing outcome data for all outcome measures. Model 7 
for course failure uses the same functional form and covariates as Model 1 but uses a binary measure of core course failure (1 = failed 
a core course, 0 = did not fail a core course) as the outcome measure instead of a binary measure of course failure. Model 7 for low GPA 
uses the same functional form and covariates as Model 1 but excludes schools and their matched pairs where GPA was reported on a 
12.0 scale. The student analytic sample for Model 1 is smaller than the total student sample by 12 observations for chronic absence, 6 
observations for low GPA, 57 observations for suspensions, and 6 observations for progress. These differences in sample sizes occurred 
because students are dropped due to perfect prediction between covariates and outcomes in the statistical analyses in Model 1. Model 
2 excludes the covariates and thus avoids losing observations due to perfect prediction during estimation. 

Source: Authors’ analysis based on extant student records described in appendix C. 
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Table D2. Results of sensitivity models with continuous versions of the outcome 
variables for chronic absence, course failure, grade point average, and progress in 
school in 2014/15 



Adjusted means 



Hedges' 

Student 

School 

Continuous specification 

EWIMS 

schools 

Control 

schools 

Standard 

error 

p-value 

g effect 
size 

sample 

size 

sample 

size 

Student risk indicators 

Chronic absence (percentage of 
instructional time missed) 

5.4 

6.5 

0.003 

.003 

-0.12 

35,888 

65 

Course failure (percentage of 
courses failed) 

8.0 

9.6 

0.005 

.004 

-0.08 

35,133 

65 

Grade point average 

2.98 

2.87 

0.041 

.006 

0.07 

30,086 

57 

Student progress in school 

Credits earned 

13.05 

13.01 

0.382 

.902 

-0.006 

35,050 

63 


Note: Estimates shown were produced using a multilevel regression model with students at level 1 and 
schools at level 2. The continuous outcomes were regressed on binary indicators for student demographic co- 
variates and grade, the grade 8 standardized tests in reading and math, missing data flags for each covariate, 
a set of dummy variables that captures school matched pairs, and the treatment indicator, which identified 
schools as EWIMS schools or control schools. Sensitivity analyses that used the continuous data instead of 
a binary flag for suspensions could not be conducted because some schools provided data on the number 
of days suspended, while others provided data on the number of suspensions. Schools did not always clarify 
whether the suspension data was reported in terms of days or incidents. 

Source: Authors' analysis based on extant student records described in appendix C. 


These analyses are displayed in table D3 and summarized as follows: 

• The average percentage of instructional time missed was significantly lower for 
students in EWIMS schools than for students in control schools. This is consis- 
tent with the main impact analysis showing that EWIMS had an impact in the 
same direction on the binary indicator of chronic absence. 

• The average percentage of courses failed out of those attempted was significantly 
lower for students in EWIMS schools than for students in control schools; this too 
is consistent with the main impact analysis showing an impact in the same direc- 
tion on the binary indicator of course failures. 

• Average GPA was statistically significantly higher for students in EWIMS schools 
than for students in control schools; this finding is inconsistent with the main 
impact analysis showing no statistically significant effect on the binary indi- 
cator of low GPA. A model excluding students at schools or the matched pairs 
of schools that used a 12.0 scale for GPA was also run to test the sensitivity of 
the results. The results of this model indicated that the adjusted mean for stu- 
dents in EWIMS schools was 2.72 and for students in control schools was 2.65 
(SE = 0.04, p-value = .07, Hedges’ g effect size = 0.07, student sample = 26,836, 
school sample = 51). 

• The average number of credits earned was not statistically significantly different 
for students in EWIMS schools than for students in controls schools; this finding 
is consistent with the main impact analyses on student progress in school. 

Detailed findings for the exploratory analyses used to test the impact of the Early Warning 
Intervention and Monitoring System for first-year and second-year students 

Results from analyses of the differential impact of EWIMS on the main student outcomes 
showed that the impact of EWIMS on chronic absence and course failure was larger for 
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Table D3. The impact of the Early Warning Intervention and Monitoring System on 
all binary outcomes for first-year and second-year students in 2014/15 




Predicted probability 







Outcome 

EWIMS 

first 

year 

Control 

first 

year 

EWIMS 

second 

year 

Control 

second 

year 

p-value 

First 

year 

sample 

Second 

year 

sample 

Total 

student 

sample 

School 

sample 

Student risk indicators 

Chronic 

absence 

0.085 

0.135 

0.126 

0.172 

.027 

18,359 

17,517 

35,876 

65 

Course 

failure 

0.246 

0.324 

0.299 

0.350 

.011 

18,114 

16,989 

35,133 

65 

Low GPA 

0.198 

0.209 

0.211 

0.231 

.292 

15,269 

14,811 

30,080 

57 

Suspended 

0.092 

0.094 

0.084 

0.088 

.559 

18,104 

17,397 

35,501 

63 

Student progress in school 

Progress in 

school 

0.124 

0.120 

0.132 

0.135 

.727 

18,025 

17,019 

35,044 

63 


EWIMS is the Early Warning Intervention and Monitoring System. GPA is grade point average. 

Note: The predicted probabilities in the treatment and control groups for students by grade were produced 
using a multilevel logistic regression model with students at level 1 and schools at level 2. Outcomes were 
regressed on binary indicators of student demographic covariates and grade, the grade 8 standardized tests 
in reading and math, missing data flags for each covariate, a set of dummy variables that capture school 
matched pairs, an interaction term between the treatment indicator and an indicator of being a second-year 
student, and an interaction term between the treatment indicator and an indicator of being a first-year student. 
The p-value corresponds to a test of the null hypothesis that these two interaction terms are not different 
or, in other words, that the treatment impact is not different for first- and second-year students. The student 
analytic samples for chronic absence, low GPA, suspended, and progress in school are the same as those 
listed for Model 1 in table D1 for each respective outcome. Less than 1 percent of students were dropped due 
to perfect prediction between covariates and outcomes in the statistical analyses. 

Source: Authors' analysis based on extant student records described in appendix C. 


first-year students than for second-year students; there were no differential effects on the 
other outcomes (see table D2). The difference between EWIMS and control schools in 
the percentage of students at risk of not graduating on time due to chronic absence was 
5.0 percentage points for first-year students and 4.6 percentage points for second-year stu- 
dents. This difference is of little practical importance (about 5 percent for both), but was 
statistically significant. The difference between EWIMS and control schools in the per- 
centage of students failing one or more courses was 7.8 percentage points for first-year stu- 
dents and 5.1 percentage points for second-year students (figure Dl). 

Supplementary analysis of the preliminary persistence measure 

Analysis of persistence in school was not considered a primary student outcome due to 
limitations of the enrollment and exit code data available for the study (see “Outcomes 
not measured or measured with limitations” in appendix C). According to the enrollment 
and exit code data available at the end of the 2014/15 school year, 95 percent of students 
in both EWIMS and control schools persisted through the end of the 2014/15 school year. 
Only about 1 percent of students were recorded as having dropped out of high school 
before the end of the 2014/15 school year, both in the EWIMS and control schools. The 
remaining 4 percent of students had unclear enrollment status (figure D2). 

Exploratory analyses that tested the difference between the percentage of students who 
were still enrolled and those who had dropped out (and excluded students with unclear 
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Figure Dl. The impacts of the Early Warning Intervention and Monitoring System on 
chronic absence and course failure were larger for first-year students than second- 
year students at the end of the 2014/15 school year 


Difference in model-adjusted percentage of students at risk between EWIMS and control schools 
10 

■ First-year students ■ Second-year students 



Chronic absence* Failed any course* 


Low GPA 


- 0.3 

Low credits 


Suspended 
Outcome 

* difference in the estimated impacts for first-year and second-year students was statistically significant at 
the p < .05 level. 

EWIMS is the Early Warning Intervention and Monitoring System. GPA is grade point average. 


Note: Sample included 65 schools and 35,876 students for chronic absence, 65 schools and 35,133 
students for failed any course, 57 schools and 30,080 students for low GPA, and 63 schools and 35,501 
students for suspended. See table D3 for more information about these sample sizes. 

Source: Authors' analysis based on extant student records from schools, school districts, and state education 
agencies described in appendix C. 


Figure D2. Percentage of students still enrolled, not enrolled, and with unclear 
enrollment status at the end of the 2014/15 school year 


Percent 

100 

75 - 

50 - 

25 - 


i Persisted in school ■ Dropped out ■ Unclear enrollment status 





Overall 


Grade 9 


Grade 10 


EWIMS is the Early Warning Intervention and Monitoring System. 

Note: The bar for control schools in grade 9 does not sum to 100 percent because of rounding. In this bar, 
97.63 percent of grade 9 students in control schools persisted, 0.61 percent did not persist, and 1.76 percent 
had an unclear enrollment status. 

Source: Authors' analysis based on extant student records described in appendix C. 
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enrollment status) showed there was no difference in preliminary persistence between 
EWIMS and control schools (table D4). Again, due to the enrollment and exit code data 
limitations, results on student preliminary persistence should be interpreted with caution. 

In addition to the impact model described in the previous paragraph, a model that 
included an interaction between the treatment condition indicator and the indicator of 
grade was analyzed. The interaction term in this model was not statistically significant 
(pwalue = .461), indicating that there was no differential impact of being in an EWIMS 
school on preliminary persistence by grade. 

Detailed findings for the exploratory school data culture analyses 

EWIMS had no detectable impact on school data culture (table D5). 

The school data culture analyses were conducted on the full sample of schools that 
responded to the school survey (n = 66). As a sensitivity check, the school data culture 
analyses were run with a restricted sample that only included matched pairs in each of the 
analytic samples (the samples for chronic absences, course failure, low GPAs, suspensions, 
and progress in school). The sensitivity checks were consistent and revealed no change 
in direction or statistical significance in the school data culture findings, confirming that 
there was no detectable impact on school data culture. 

Detailed findings on satisfaction with Early Warning Intervention and Monitoring System training 

Satisfaction surveys were collected after each training to document satisfaction with EWIMS 
implementation. Overall, satisfaction was high across all trainings. More than 90 percent of 
respondents were either satisfied or very satisfied with each training (table D6). 

Detailed findings for the level of implementation 

Only two schools achieved moderate or high overall implementation. However, some 
schools achieved moderate or high scores for individual steps or on individual indica- 
tors within steps (table D7). More than 50 percent of schools achieved high ratings on 


Table D4. The impact of the Early Warning Intervention and Monitoring System on 
preliminary persistence in the 2014/15 school year 


Predilcted 
probability 
for EWIMS 

Predicted 
probability 
for control 

Odds ratio 

Standard 

error 

p-value 

Cox index 
effect size 

Student 

sample 

size 

School 

sample 

size 

0.99 

0.99 

0.83 

0.10 

.123 

-0.11 

33,124 

59 


EWIMS is the Early Warning Intervention and Monitoring System. 

Note: Estimates shown were produced using a multilevel logistic regression model with students at level 1 and 
schools at level 2. The binary indicator of preliminary persistence was regressed on binary indicators for stu- 
dent grade, a set of dummy variables that captures school matched pairs, and the treatment indicator, which 
identified schools as EWIMS schools or control schools. This model specification does not match that of the 
primary impact models (see table Dl) because the model that used preliminary persistence as an outcome 
did not converge when additional student demographic covariates and student test scores were included due 
to perfect prediction. 

Source: Authors' analysis based on extant student records described in appendix C. 
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Table D5. The impact of the Early Warning Intervention and Monitoring System on 
school data culture during 2014/15 




EWIMS schools 


Control schools 

Standardized 


Characteristic 

n 

Mean 

Standard 

deviation 

n 

Mean 

Standard 

deviation 

difference 
(Hedges g) a 

p-value 

Overall data culture 

32 

2.92 

0.44 

34 

2.81 

0.37 

0.27 

.301 

Dimensions of data culture 

Context for data use 

33 

2.74 

0.41 

34 

2.75 

0.38 

-0.02 

.947 

Concrete supports for 

data use 

32 

2.85 

0.53 

34 

2.74 

0.46 

0.22 

.418 

Data-driven student 









support 

32 

3.06 

0.55 

34 

2.90 

0.47 

0.31 

.196 

Lack of barriers to 

data use b 

32 

2.82 

0.56 

34 

2.72 

0.46 

0.19 

.424 


EWIMS is the Early Warning Intervention and Monitoring System, n is the number of schools. 

Note: Sample included 66 schools that completed the school leader surveys. Data culture items were mea- 
sured on a l-to-4 scale, with 1 being low data culture and 4 being high data culture. Regression models that 
regressed data culture on treatment status, a set of three covariates (school size, baseline on-time graduation 
rate, and baseline data-driven dropout prevention efforts), and a set of variables capturing school matched 
pairs revealed no statistically significant differences at the p < .05 level. 

a. Hedges' g is used as the standardized difference to account for small sample sizes. None of the differ- 
ences was statistically significant at the p < .05 level. 

b. The items that compose the scale for barriers to data use were reverse coded, such that a higher score 
indicated fewer barriers. 

Source: Authors' analysis based on school leader survey administered in spring 2015. 



Table D6. Participant satisfaction with Early Warning Intervention and Monitoring 
System trainings during 2013/14 and 2014/15 

Not satisfied 

Satisfied 

Very satisfied 

Training Number 

Percent Number 

Percent 

Number 

Percent 

Regional training 4 

2.3 

99 

57.2 

38 

39.3 

Early warning data tool training 1 

1.3 

27 

34.6 

50 

64.1 

Early warning data tool refresher 3 

6.7 

13 

28.9 

29 

64.4 

Site Visit 1 2 

1.6 

44 

34.1 

82 

63.5 

Site Visit 2 0 

0.0 

31 

37.0 

52 

63.0 

Site Visit 3 3 

3.0 

41 

38.0 

64 

59.0 

WebShare 1 3 

8.6 

25 

71.4 

7 

20.0 

WebShare 2 1 

4.3 

15 

65.2 

7 

30.4 

WebShare 3 1 

4.2 

14 

58.3 

9 

37.5 

WebShare 4 2 

9.5 

10 

47.6 

9 

42.9 

WebShare 5 0 

0.0 

12 

52.2 

11 

47.8 


Note: No respondents chose the lowest option, ‘Not at all satisfied,’ on any of the satisfaction surveys. Two 
participants did not respond to the satisfaction item for regional training, and one participant did not respond 
to the satisfaction item for site visit 1; therefore, the percentages do not add to 100 percent. 

Source: Authors' analysis based on attendance sheets from the 2013/14 and 2014/15 trainings. 
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six indicators, including 89 percent of schools on the indicator related to EWIMS team 
composition. 

Although few schools achieved high overall implementation in the 2014/15 school year, 
many schools achieved high implementation on individual steps (table D8). Seventy 
percent of schools achieved high implementation on at least one step, and 24 percent of 
schools achieved high implementation on at least four steps. 

Supplementary findings for the barriers experienced by schools that stopped or never began 
implementing the Early Warning Intervention and Monitoring System 

Eight schools stopped or never began implementing EWIMS during the 2014/15 school 
year. Exit interviews were conducted with seven of these schools. In these interviews, 
school staff cited issues related to the early warning data tool as a reason to stop imple- 
mentation. Staff from four schools said the tool took too long to use, and two schools said 
they could not access the tool at all. In addition, three schools reported that staff turnover 


Table D7. Percentage of Early Warning Intervention and Monitoring System schools 
that achieved low, moderate, or high implementation ratings during 2014/15, by 
indicator 



Low score 

Moderate score 

High score 

Indicator 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Overall Implementation 

35 

95 

1 

3 

1 

3 

Step 1 — Overall 

16 

43 

7 

19 

14 

38 

Step 1.1 — Champion 

13 

35 

9 

24 

15 

41 

Step 1.2 — Team membership 

2 

5 

2 

5 

33 

89 

Step 1.3 — Meeting frequency 

10 

27 

16 

43 

11 

30 

Step 2 — Overall 

6 

16 

12 

32 

19 

51 

Step 2.1 — Timing of imports 

4 

11 

6 

16 

27 

73 

Step 2.2 — Correct and complete data 

4 

11 

5 

14 

28 

76 

Step 2.3 — Tracking interventions 

6 

16 

9 

24 

22 

59 

Step 3 — Overall 

18 

49 

15 

41 

4 

11 

Step 3.1 — Identification of at-risk students 

6 

16 

7 

19 

24 

65 

Step 3.2 — Frequency of student data review 

11 

30 

15 

41 

11 

30 

Step 3.3 — Reviewing patterns of risk 

14 

38 

12 

32 

11 

30 

Step 4 — Overall 

8 

22 

22 

59 

7 

19 

Step 4.1 — Introduce supplemental data 

6 

16 

5 

14 

26 

70 

Step 4.2 — Understanding underlying causes 

8 

22 

21 

57 

8 

22 

Step 5 — Overall 

21 

57 

8 

22 

8 

22 

Step 5.1 — Inventory of interventions 

2 

5 

17 

46 

18 

49 

Step 5.2 — Assignment of interventions — frequency 

5 

14 

15 

41 

17 

46 

Step 5.3 — Assignment of interventions — coverage 

20 

54 

17 

46 

0 

0 

Step 6 — Overall 

21 

57 

10 

27 

6 

16 

Step 6.1 — Track student progress in interventions 

15 

41 

8 

22 

14 

38 

Step 6.2 — Monitor effectiveness of interventions 

17 

46 

12 

32 

8 

22 

Step 7 — Overall/Reflection on team progress 

8 

22 

11 

30 

18 

49 


Note: School n = 37, including one school that never implemented EWIMS. 

Source: Authors' calculations based on school leader survey, early warning data tool reports, monthly meeting 
logs, and EWIMS team interviews. 
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Table D8. Number of steps on which Early Warning Intervention and Monitoring 
System schools achieved high implementation ratings during 2014/15 


1 Number of steps on which school received “high rating” 

Number of schools 

Percent 1 

Zero 

11 

30 

One 

3 

8 

Two 

9 

24 

Three 

5 

14 

Four 

6 

16 

Five 

2 

5 

Six 

1 

3 


Note: School n = 37, including one school that never implemented the Early Warning Intervention and Monitor- 
ing System. 

Source: Authors' calculations based on school leader survey, early warning data tool reports, monthly meeting 
logs, and Early Warning Intervention and Monitoring System team interviews. 


contributed to their decision to stop implementing EWIMS, given that they were unable 
to find replacements for key EWIMS team members. Three schools also said they stopped 
implementing EWIMS in part because of other school' or district-based initiatives that 
they perceived to be competing with EWIMS for time and resources. For example, one 
school identified a new technology initiative as a project that drew technology staff away 
from supporting EWIMS, while another school said schoohbased administrators had to 
devote time to a new educator evaluation system. 

Detailed findings for the percentage of students flagged in the early warning tools 

For a variety of reasons, the percentages of students flagged in the early warning tools 
(described on page 14 in the main report) differ from those presented in figure 3 in the 
main report. First, the percentages reported are based on different samples. In figure 3, 
the students are those in grades 9 or 10, in either the EWIMS or control group analytic 
samples, who have complete outcome data. The number of flagged students includes all 
students in EWIMS schools with data uploaded into the tool, and there were more schools 
with data in the tools than were included in the impact analyses. Second, the percentag' 
es presented in figure 3 are regression-adjusted predicted probabilities from models that 
include covariates, while on page 14, they are raw percentages. Third, figure 3 pres- 
ents the percentage of students with each of the indicators across the full year, whereas 
on page 14, the percentages represent the proportion of students ever flagged over the 
course of the year in the early warning data tools. This may explain the higher rates of 
students flagged for attendance and course performance on page 14 than in figure 3. 

Detailed findings for the specific types of interventions offered in Early Warning Intervention and 
Monitoring System schools 

To document interventions used in EWIMS schools, researchers extracted and coded 
information from three different data sources: the school leader surveys from spring 2014 
and 2015, the early warning data tool reports, and the EWIMS team interviews. Academic 
interventions of any type, tutoring, and traditional credit recovery coursework were most 
common among EWIMS schools (table D9). 
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Table D9. Number and percentage of Early Warning Intervention and Monitoring 
System schools that reported having different interventions and supports available 
for students in the 2014/15 school year 


EWIMS schools that reported 

Intervention was available 

Intervention type 

Number 

Percent 

English language arts support 

14 

38 

Algebra support 

13 

35 

Targeted academic support 

12 

32 

Other math support 

9 

24 

Remediation 

2 

5 

Tutoring 

25 

68 

Afterschool tutoring 

13 

35 

Peer tutoring 

7 

19 

Credit recovery 

24 

65 

Online credit recovery 

10 

27 

Mentoring 

23 

62 

Peer mentoring 

11 

30 

Check and connect 

3 

8 

Meet with students and parents 

15 

41 

Letter or phone call home 

14 

38 

Monitoring 

13 

35 

Counseling 

11 

30 

Student contracts 

9 

24 

Mental and physical health services 

9 

24 

Mental health 

5 

14 

Connecting students to community resources 

4 

11 

Teen pregnancy 

2 

5 

Truancy actions 

9 

24 

Disciplinary action (detention, demerits) 

6 

16 

Freshman transition programs 

7 

19 

Social emotional interventions 

5 

14 

Homework or study space interventions 

4 

11 

Alternative education 

3 

8 

Response to intervention 

3 

8 

Teaching and instruction interventions 

2 

5 

Generic academic intervention 

26 

70 

Other 

26 

70 


EWIMS is the Early Warning Intervention and Monitoring System. 

Source: Authors' calculations based on school leader survey, early warning data tool reports, and EWIMS team 
interviews. 


Detailed findings for the treatment contrast analyses 

The study used end-of-year survey data to examine the contrast between EWIMS and 
control schools in the practices they used to identify and support students at risk of not 
graduating on time. The surveys captured the extent to which schools adhered to their 
randomly assigned groups (according to self-reports of whether they used an early warning 
system). The surveys also measured whether schools had a dedicated team to identify and 
support at-risk students, the frequency of data review, and the number and type of inter- 
ventions offered to students (see appendix C). 
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Use of an early warning system. Most EWIMS schools (27) and a few control schools (4) 
reported using an early warning system to organize their data review and assign students 
to interventions during the 2014/15 school year (table DIO). EWIMS schools were signifi- 
cantly more likely than control schools to report using an early warning system. Of the 
66 schools that responded to this item on the school leader survey, 87 percent of EWIMS 
schools and 10 percent of control schools reported using an early warning system in the 
2014/15 school year (see figure 7 in the main report and table DIO). 

Dedicated team to identify and support at-risk students. EWIMS schools were more 
likely than control schools to report having a dedicated team to identify and support 
at-risk students (see figure 7 in the main report and table DIO). Three-quarters (76 percent) 
of EWIMS schools and 40 percent of control schools reported that they had a dedicated 
team to identify and support at-risk students. 

Frequency of data review. Both EWIMS and control schools reported reviewing atten- 
dance and course failure data frequently (table Dll). 


Table DIO. The percentages of schools that used an early warning system and had 
a dedicated school-based team differed by treatment status during 2014/15 


Outcome 

EWIMS 

schools 

Control 

school 

Odds 

ratio 

Cox index 
effect size 

p-value 

Use of an early warning system 

87 

10 

60.78 

2.49 

<.001 

Use of a dedicated team to identify and support 

at-risk students 

76 

40 

4.77 

0.95 

.006 


EWIMS is the Early Warning Intervention and Monitoring System. 

Note: The sample for the early warning system item included 66 schools that responded to this item on the 
school leader surveys (32 EWIMS schools and 34 control schools). The sample for the “dedicated team to 
identify and support at-risk students” item included the 65 schools that responded to this item on the school 
leader survey (31 EWIMS and 34 control schools). The percentages represent predicted probabilities from 
logistic regression models that regressed a binary indicator of whether or not a school used an early warning 
system or had a dedicated team to identify and support at-risk students on treatment status and a set of 
three covariates (school size, baseline on-time graduation rate, and baseline data-driven dropout prevention 
efforts). The odds ratios and p-values come from these same regression models. 

Source: Authors' analysis of school leader survey. 


Table Dll. Frequency of attendance and course failure data review, as reported on the 
school leader survey at the end of the 2014/15 school year (percentage of schools) 

Outcome 

Weekly 
or more 

often 

Monthly 

4 times 
per year 

3 times 
per year 

2 times 
or less 
per year 

Attendance review in EWIMS schools 

38 

38 

14 

7 

3 

Attendance review in control schools 

38 

48 

10 

0 

3 

Course failure data review in EWIMS schools 

21 

31 

31 

7 

10 

Course failure data review in control schools 

19 

26 

35 

6 

13 


EWIMS is the Early Warning Intervention and Monitoring System. 

Note: Sample includes 60 schools that responded to these two items on the school leader survey (29 EWIMS 
and 31 control schools). 

Source: Authors’ calculations based on school leader survey. 
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Table D12. Statistical analyses of the frequency of attendance and course failure 
data review between Early Warning Intervention and Monitoring System and control 
schools, as reported on the school leader survey at the end of the 2014/15 school 
year 


Outcome 

EWIMS 

schools 

(percentage) 

Control 

school 

(percentage) 

Odds ratio 

Cox index 
effect size 

p-value 

Reviewed attendance 






data at least monthly 

76 

88 

0.44 

-0.50 

.248 

Reviewed course failure 






data at least monthly 

53 

42 

1.52 

0.25 

.454 


EWIMS is the Early Warning Intervention and Monitoring System. 

Note: Sample includes 60 schools that responded to these two items on the school leader survey (29 EWIMS 
and 31 control schools). The percentages represent predicted probabilities from logistic regression models 
that regressed a binary indicator of whether or not a school reviewed data at least monthly on treatment 
status and a set of three covariates (school size, baseline on-time graduation rate, and baseline data-driven 
dropout prevention efforts). The odds ratios and p-values come from the same regression models. 

Source: Authors' calculations based on school leader survey. 


Models that tested the difference between the percentage of EWIMS and control schools 
that reviewed attendance and course failure data at least monthly revealed no statistically 
significant difference in the frequency of data review (table D12). 


Number of interventions. There were no statistically significant differences between 
EWIMS schools and control schools in the reported number of interventions available 
to support students (EWIMS schools mean [M] = 2.75, standard deviation [SD] = 2.21; 
control schools [M] = 2.20, [SD] = 2.20, (3 = 0.55, SE = 0.58, p = .358, Hedges’ g = 0.29). 


Type of interventions. The study examined whether EWIMS schools were more likely 
than control schools to offer at least one intervention in each of the three domains in 
which students can be flagged as at risk: chronic absence, course failure, and behavior. 
Survey responses indicated that a similar percentage of EWIMS and control schools offered 
interventions in each domain (table D13). Also, because there were so few attendance and 


Table D13. The number of schools offering each type of intervention, by condition, 


during the 2014/15 school year 

Outcome 

EWIMS 

schools 

(percentage) 

Control 

school 

(percentage) 

Odds ratio 

Cox index 
effect size 

p-value 

Attendance intervention 

11 

9 

1.18 

0.10 

.850 

Behavioral intervention 

16 

9 

1.85 

0.37 

.459 

Course performance intervention 

96 

100 

0.00 

na 

.997 


na is not applicable because the Cox index effect size was not calculated for course performance interven- 
tions, given that the odds ratio was zero and the natural log of the odds ratio (used to calculate the Cox index 
effect size) would create an uninterpretable value. 

Note: Sample includes 59 schools that responded to these items on the school leader survey (28 EWIMS and 
31 control schools). The percentages represent predicted probabilities from logistic regression models that 
regressed a binary indicator of whether or not a school offered each type of intervention on treatment status 
and a set of three covariates (school size, baseline on-time graduation rate, and baseline data-driven dropout 
prevention efforts). The odds ratios and p-values come from these same regression models. 

Source: Authors' analysis of school leader survey. 
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behavior interventions reported in the school survey, only five schools offered at least one 
intervention in all three domains and there was no difference between the number of 
EWIMS and control schools that offered supports for students in all three domains. 

The treatment contrast analyses were conducted on the full sample of schools that 
responded to the school survey (n = 66). As a sensitivity check, models were also run with 
a restricted sample that only included matched pairs (for chronic absence, course failure, 
low GPAs, suspensions, and progress in school) in each of the analytic samples. The sensi- 
tivity checks were consistent and revealed no change in direction or statistical significance 
in the treatment contrast analyses. 
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Appendix E. Disclosure of potential conflicts of interest 


The following section discloses any potential conflicts of interest for the Regional Educa- 
tional Laboratory (REL) Midwest study team in its evaluation of the Early Warning Inter- 
vention and Monitoring System (EWIMS). EWIMS was originally developed by American 
Institutes for Research under a cooperative agreement with the U.S. Department of Edu- 
cation to operate the National High School Center. American Institutes for Research 
conducted the evaluation of EWIMS as part of its work with REL Midwest’s Dropout Pre- 
vention Research Alliance. 

To conduct the study, REL Midwest established a conservative firewall between implemen- 
tation of the EWIMS model and its evaluation through the use of two separate teams: an 
implementation team and an evaluation team. The evaluation team collected all outcome 
data and analyzed all data for the report. The implementation team worked directly with 
the EWIMS schools to implement the intervention, including providing training and 
technical assistance. The implementation team did help facilitate data collection in the 
EWIMS schools, such as collecting sign-in sheets at professional development sessions, 
providing links to satisfaction surveys, and reminding school staff to submit administra- 
tive data to the evaluation team. However, the implementation team was not involved 
in any data collection in control schools nor in any analysis or reporting tasks. Further- 
more, the evaluation team did not share any data or results with the implementation team 
during the implementation period other than the data that the team would have collected 
during typical implementation of EWIMS (for example, results from post-training satis- 
faction surveys). The evaluation team did not share any findings from the impact analyses 
with the implementation team until the implementation period for the year of evaluation 
(2014/15) had ended. 

In addition, the report authors and their potential conflicts of interest are noted below. 

Ann-Marie Faria, Ph.D. She is the co-Project Director of this study and does not have any 
notable conflicts of interest. 

Nicholas Sorensen, Ph.D. He is the co-Project Director of this study and does not have any 
notable conflicts of interest. 

Jessica Heppen, Ph.D. She is the Principal Investigator of this study. Through her work 
with the National High School Center, Dr. Heppen was involved in designing the original 
versions of the Early Warning Intervention and Monitoring System, which was the basis of 
the interventions examined in this study. Dr. Heppen, who has designed and conducted a 
number of experimental evaluation studies, was also involved in the design of the current 
study and in contextualizing the findings in the broader literature. However, Dr. Heppen 
was not directly involved in any data collection or analysis. 

Jill Bowdon, Ph.D. She is the lead analyst of this study and does not have any notable con- 
flicts of interest. 

Suzanne Taylor. She is an analyst on this study, and she does not have any notable conflicts 
of interest. 
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Ryan Eisner. He is an analyst on this study, and he does not have any notable conflicts of 
interest 

Shandu Foster. He is a research associate on this study, and he does not have any notable 
conflicts of interest. 
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Notes 


The study team would like to acknowledge the contributions of the members of the Dropout 
Prevention Research Alliance for their ongoing support and guidance. In addition, for 
dedication to delivering the Early Warning Intervention and Monitoring System in pan 
ticipating high schools, the study team thanks the technical assistance liaisons, including 
Mindee O’Cummings, Amy Szymanski, David Blumenthal, Jenny Scala, Bersheril Bailey, 
Deborah Gurke, Marie Husby-Slater, Traci Karageorge, Helen Muhisani, Mara Schanfield, 
Wendy Surr, Matthew Welch, and Montrischa Williams. The study leaders would like to 
acknowledge the efforts of team members who supported the evaluation over the three- 
year study, including Dionisio Garcia-Perez, Laura Checovich, Lily Heine, Elizabeth Davis, 
Patricia Balana, and Luke Keele. The study leaders also thank Mike Garet, Jordan Rickies, 
and Susan Therriault for their expert review of the report. Linally, the study team thanks 
the staff in the many schools that participated in the study. Without their commitment, 
this project would not have been feasible. 

The following are past and present members of the Midwest Dropout Prevention Research 
Alliance: Mary Barrie, Ed.D., Minnesota Department of Education; Teresa Brown, Indiana 
Department of Education; Thomas Cooley, Iowa Department of Education; Steve Crew, 
Iowa Department of Education; Walter DeBoer, Grand Rapids Public Schools; Leisa Gal- 
lagher, State Superintendent’s Dropout Prevention Challenge, Michigan Department of 
Education (former member); John Gimpl, Minnesota Department of Education; Jared 
Knowles, Wisconsin Department of Public Instruction; Laurie Kruszynski, Toledo Public 
Schools; Harold Sweeney, Boone-Winnebago Regional Office of Education; Craig Weth- 
ington, Minnesota Department of Education; and Sue Zake, Ph.D., Ohio Department of 
Education. 

1. The off-track flag definition is based on Allensworth and Easton’s (2005, 2007) work 
on the on-track indicator. 

2. National on-time graduation statistics are from 2013/14, with the exception of sub- 
group data, which are from 2011/12. 

3. Early warning indicators and systems focus on on-time graduation, that is, gradu- 
ating within four years of entering high school with a regular diploma, as the goal 
(Allensworth & Easton, 2007; Heppen & Therriault, 2008; Stetser & Stillwell, 2014). 

4. “Data culture” refers to the ways in which schools use data to make decisions and iden- 
tify students in need of additional support. 

5. The final sample of 73 schools was larger than the intended sample of 72 because the 
marginal cost of including the last school was relatively low (see table B1 in appendix B). 

6. Sensitivity analyses that used the continuous data instead of a binary flag for suspen- 
sions could not be conducted because some schools provided data on the number of 
days suspended, while others provided data on the number of suspensions; 22 schools 
did not clarify which type of data were provided. 

7. An effect size is a standardized metric for reporting the magnitude of differences 
between two groups, expressed in terms of the number of standard deviations (for 
example, EWIMS schools were 0.27 standard deviations higher in school data culture 
than control schools). The What Works Clearinghouse (2014) considers effect sizes of 
0.25 standard deviations or larger to be substantively important. That is, effect sizes of 
0.25 or larger are interpreted as a meaningful effect, even though they may not reach 
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statistical significance in a given study. Hedges’ g effect sizes use a pooled standard 
deviation across treatment and control schools. 

8. The eight schools that stopped or never began implementing EWIMS in the 2014/15 
school year were not considered to have formally dropped out of the study. Schools 
were only considered to have attrited from the study if they stopped participating in 
the student'level data collection or if their matched pair counterpart did not partici- 
pate in the student-level data collection. Six of the eight schools that stopped or never 
began implementing EWIMS continued to participate in the data collection and were 
included in the study samples for the main impact analyses. See appendix D for a dis- 
cussion of the reasons why these schools stopped implementing EWIMS. 

9. Although the total number of EWIMS schools included in the impact analyses varied 
from 29 to 33 (depending on the outcome), the two schools that achieved moderate or 
high quality implementation were included in all analyses. The number of low-imple- 
menting EWIMS schools ranged from 27 to 31 (93-94 percent of the analytic samples 
for the different student outcomes in the impact analyses). 

10. This may not indicate a problem with implementing EWIMS; rather, it may indicate 
that schools’ own data systems provided all of the capabilities of the early warning 
data tool they were offered. 

11. The number of students with uploaded data is larger than the size of the analytic 
sample for the EWIMS schools. This is because two EWIMS schools that did not 
provide outcome data for the study did upload student data into their tools during 
implementation. 

12. There were 20 schools that received a score of 1 (27 percent), 11 schools that received 
a score of 2 (15 percent), 32 schools that received a score of 3 (44 percent), and 10 
schools that received a score of 4 (14 percent). 

13. Although eight EWIMS schools stopped or never began implementing EWIMS during 
the study, they were not considered to have formally withdrawn from the study if they 
continued to provide outcome data. Only two of the eight schools also formally with- 
drew from the study and did not provide outcome data; the remaining six were includ- 
ed in the analytic sample. 

14. The study team also collected these data to inform intervention analyses at the end of 
the 2013/14 school year, before EWIMS implementation began. 

15. In spring 2014, the survey response rate was 97 percent (71 schools). 

16. Item difficulty reflects how positively an item is endorsed. Items with low item dif- 
ficulty will be frequently and positively endorsed (for example, a high frequency of 
“Strongly Agree”). 

17. Students with exit codes of homeschooled were coded as having an unclear enroll- 
ment status because states and districts reported using this code for students who were 
under the state compulsory school attendance age (age 18 in all three states) but whose 
parents or guardians had formally withdrawn them from schooling. 

18. In large sample sizes (those approaching infinity), Cohen’s d and Hedges’ g will produce 
the same standardized difference value. However, for smaller samples sizes, Hedges’ g 
uses N - 1 in the variance estimate (as opposed to the N used in Cohen’s d calcula- 
tions, which underestimates the variance and therefore overestimates the standardized 
difference). 

19. Interviews included exit interviews with schools that stopped or never began imple- 
menting EWIMS in the 2014/15 school year. 

20. As mentioned, the 2014 surveys were only used to gather information on the interven- 
tions used in EWIMS schools. 
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